You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2022/05/01 13:30:51 UTC
[GitHub] [drill] cgivre opened a new pull request, #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
cgivre opened a new pull request, #2526:
URL: https://github.com/apache/drill/pull/2526
# [DRILL-8204](https://issues.apache.org/jira/browse/DRILL-8204): Allow Provided Schema for HTTP Plugin in JSON Mode
## Description
See below. 👇
## Documentation
One of the challenges of querying APIs is inconsistent data. Drill allows you to provide a schema for individual endpoints. You can do this in one of two ways: either by
providing a serialized TupleMetadata of the desired schema. This is an advanced functionality and should only be used by advanced Drill users.
The schema provisioning currently supports complex types of Arrays and Maps at any nesting level.
### Example Schema Provisioning:
```json
"jsonOptions": {
"providedSchema": [
{
"fieldName": "int_field",
"fieldType": "bigint"
}, {
"fieldName": "jsonField",
"fieldType": "varchar",
"properties": {
"drill.json-mode":"json"
}
},{
// Array field
"fieldName": "stringField",
"fieldType": "varchar",
"isArray": true
}, {
// Map field
"fieldName": "mapField",
"fieldType": "map",
"fields": [
{
"fieldName": "nestedField",
"fieldType": "int"
},{
"fieldName": "nestedField2",
"fieldType": "varchar"
}
]
}
]
}
```
### Example Provisioning the Schema with a JSON String
```json
"jsonOptions": {
"jsonSchema": "{\"type\":\"tuple_schema\",\"columns\":[{\"name\":\"outer_map\",\"type\":\"STRUCT<`int_field` BIGINT, `int_array` ARRAY<BIGINT>>\",\"mode\":\"REQUIRED\"}]}"
}
```
You can print out a JSON string of a schema with the Java code below.
```java
TupleMetadata schema = new SchemaBuilder()
.addNullable("a", MinorType.BIGINT)
.addNullable("m", MinorType.VARCHAR)
.build();
ColumnMetadata m = schema.metadata("m");
m.setProperty(JsonLoader.JSON_MODE, JsonLoader.JSON_LITERAL_MODE);
System.out.println(schema.jsonString());
```
This will generate something like the JSON string below:
```json
{
"type":"tuple_schema",
"columns":[
{"name":"a","type":"BIGINT","mode":"OPTIONAL"},
{"name":"m","type":"VARCHAR","mode":"OPTIONAL","properties":{"drill.json-mode":"json"}
}
]
}
```
## Dealing With Inconsistent Schemas
One of the major challenges of interacting with JSON data is when the schema is inconsistent. Drill has a `UNION` data type which is marked as experimental. At the time of
writing, the HTTP plugin does not support the `UNION`, however supplying a schema can solve a lot of those issues.
### Json Mode
Drill offers the option of reading all JSON values as a string. While this can complicate downstream analytics, it can also be a more memory-efficient way of reading data with
inconsistent schema. Unfortunately, at the time of writing, JSON-mode is only available with a provided schema. However, future work will allow this mode to be enabled for
any JSON data.
#### Enabling JSON Mode:
You can enable JSON mode simply by adding the `drill.json-mode` property with a value of `json` to a field, as shown below:
```json
{
"fieldName": "jsonField",
"fieldType": "varchar",
"properties": {
"drill.json-mode": "json"
}
}
```
## Testing
Added unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] vvysotskyi commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r863046227
##########
contrib/storage-http/JSON_Options.md:
##########
@@ -0,0 +1,125 @@
+# JSON Options and Configuration
+
+Drill has a collection of JSON configuration options to allow you to configure how Drill interprets JSON files. These are set at the global level, however the HTTP plugin
+allows you to configure these options individually per connection and override the Drill defaults. The options are:
+
+* `allowNanInf`: Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`: By default, Drill attempts to infer data types from JSON data. If the data is malformed, Drill may throw schema change exceptions. If your data is
+ inconsistent, you can enable `allTextMode` which when true, Drill will read all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`: By default Drill will attempt to interpret integers, floating point number types and strings. One challenge is when data is consistent, Drill may
+ throw schema change exceptions. In addition to `allTextMode`, you can make Drill less sensitive by setting the `readNumbersAsDouble` to `true` which causes Drill to read all
+ numeric fields in JSON data as `double` data type rather than trying to distinguish between ints and doubles.
+* `enableEscapeAnyChar`: Allows a user to escape any character with a \
+* `skipMalformedRecords`: Allows Drill to skip malformed records and recover without throwing exceptions.
+* `skipMalformedDocument`: Allows Drill to skip entire malformed documents without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection configuration as shown below:
+
+```json
+
+"jsonOptions": {
+ "allTextMode": true,
+ "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data. Drill allows you to provide a schema for individual endpoints. You can do this in one of three ways:
+
+1. By providing a schema inline [See: Specifying Schema as Table Function Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema. This is an advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+ "providedSchema": [
Review Comment:
Can we use fields, schema serialization, and deserialization logic from TupleMetadata? It has a good human-readable string representation that is much more compact than JSON
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
cgivre commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r863050797
##########
contrib/storage-http/JSON_Options.md:
##########
@@ -0,0 +1,125 @@
+# JSON Options and Configuration
+
+Drill has a collection of JSON configuration options to allow you to configure how Drill interprets JSON files. These are set at the global level, however the HTTP plugin
+allows you to configure these options individually per connection and override the Drill defaults. The options are:
+
+* `allowNanInf`: Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`: By default, Drill attempts to infer data types from JSON data. If the data is malformed, Drill may throw schema change exceptions. If your data is
+ inconsistent, you can enable `allTextMode` which when true, Drill will read all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`: By default Drill will attempt to interpret integers, floating point number types and strings. One challenge is when data is consistent, Drill may
+ throw schema change exceptions. In addition to `allTextMode`, you can make Drill less sensitive by setting the `readNumbersAsDouble` to `true` which causes Drill to read all
+ numeric fields in JSON data as `double` data type rather than trying to distinguish between ints and doubles.
+* `enableEscapeAnyChar`: Allows a user to escape any character with a \
+* `skipMalformedRecords`: Allows Drill to skip malformed records and recover without throwing exceptions.
+* `skipMalformedDocument`: Allows Drill to skip entire malformed documents without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection configuration as shown below:
+
+```json
+
+"jsonOptions": {
+ "allTextMode": true,
+ "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data. Drill allows you to provide a schema for individual endpoints. You can do this in one of three ways:
+
+1. By providing a schema inline [See: Specifying Schema as Table Function Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema. This is an advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+ "providedSchema": [
Review Comment:
You can sort of do that now. The issue I ran into with the `TupleMetadata` representation is that it gets weird when you have nested fields. The current implementation allows you to either use the simplified implementation or you can pass a serialized `TupleMetadata` object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] vvysotskyi commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r863495712
##########
contrib/storage-http/JSON_Options.md:
##########
@@ -0,0 +1,125 @@
+# JSON Options and Configuration
+
+Drill has a collection of JSON configuration options to allow you to configure how Drill interprets JSON files. These are set at the global level, however the HTTP plugin
+allows you to configure these options individually per connection and override the Drill defaults. The options are:
+
+* `allowNanInf`: Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`: By default, Drill attempts to infer data types from JSON data. If the data is malformed, Drill may throw schema change exceptions. If your data is
+ inconsistent, you can enable `allTextMode` which when true, Drill will read all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`: By default Drill will attempt to interpret integers, floating point number types and strings. One challenge is when data is consistent, Drill may
+ throw schema change exceptions. In addition to `allTextMode`, you can make Drill less sensitive by setting the `readNumbersAsDouble` to `true` which causes Drill to read all
+ numeric fields in JSON data as `double` data type rather than trying to distinguish between ints and doubles.
+* `enableEscapeAnyChar`: Allows a user to escape any character with a \
+* `skipMalformedRecords`: Allows Drill to skip malformed records and recover without throwing exceptions.
+* `skipMalformedDocument`: Allows Drill to skip entire malformed documents without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection configuration as shown below:
+
+```json
+
+"jsonOptions": {
+ "allTextMode": true,
+ "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data. Drill allows you to provide a schema for individual endpoints. You can do this in one of three ways:
+
+1. By providing a schema inline [See: Specifying Schema as Table Function Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema. This is an advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+ "providedSchema": [
Review Comment:
It is not necessary to have only JSON-supported types because Drill should be able to do the auto conversion from one type to another one. `TupleMetadata` has also JSON representation (see https://drill.apache.org/docs/create-or-replace-schema/#creating-a-schema), so at least this format can be used instead of the custom one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
cgivre commented on PR #2526:
URL: https://github.com/apache/drill/pull/2526#issuecomment-1115142541
@vvysotskyi Thanks for your comments and assistance! I got this to work and the HTTP plugin now supports inline schema!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
cgivre commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r863277815
##########
contrib/storage-http/JSON_Options.md:
##########
@@ -0,0 +1,125 @@
+# JSON Options and Configuration
+
+Drill has a collection of JSON configuration options to allow you to configure how Drill interprets JSON files. These are set at the global level, however the HTTP plugin
+allows you to configure these options individually per connection and override the Drill defaults. The options are:
+
+* `allowNanInf`: Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`: By default, Drill attempts to infer data types from JSON data. If the data is malformed, Drill may throw schema change exceptions. If your data is
+ inconsistent, you can enable `allTextMode` which when true, Drill will read all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`: By default Drill will attempt to interpret integers, floating point number types and strings. One challenge is when data is consistent, Drill may
+ throw schema change exceptions. In addition to `allTextMode`, you can make Drill less sensitive by setting the `readNumbersAsDouble` to `true` which causes Drill to read all
+ numeric fields in JSON data as `double` data type rather than trying to distinguish between ints and doubles.
+* `enableEscapeAnyChar`: Allows a user to escape any character with a \
+* `skipMalformedRecords`: Allows Drill to skip malformed records and recover without throwing exceptions.
+* `skipMalformedDocument`: Allows Drill to skip entire malformed documents without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection configuration as shown below:
+
+```json
+
+"jsonOptions": {
+ "allTextMode": true,
+ "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data. Drill allows you to provide a schema for individual endpoints. You can do this in one of three ways:
+
+1. By providing a schema inline [See: Specifying Schema as Table Function Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema. This is an advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+ "providedSchema": [
Review Comment:
One more thing to mention. The way I implemented it only supports the data types that are supported by the JSON reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
cgivre commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r863050797
##########
contrib/storage-http/JSON_Options.md:
##########
@@ -0,0 +1,125 @@
+# JSON Options and Configuration
+
+Drill has a collection of JSON configuration options to allow you to configure how Drill interprets JSON files. These are set at the global level, however the HTTP plugin
+allows you to configure these options individually per connection and override the Drill defaults. The options are:
+
+* `allowNanInf`: Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`: By default, Drill attempts to infer data types from JSON data. If the data is malformed, Drill may throw schema change exceptions. If your data is
+ inconsistent, you can enable `allTextMode` which when true, Drill will read all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`: By default Drill will attempt to interpret integers, floating point number types and strings. One challenge is when data is consistent, Drill may
+ throw schema change exceptions. In addition to `allTextMode`, you can make Drill less sensitive by setting the `readNumbersAsDouble` to `true` which causes Drill to read all
+ numeric fields in JSON data as `double` data type rather than trying to distinguish between ints and doubles.
+* `enableEscapeAnyChar`: Allows a user to escape any character with a \
+* `skipMalformedRecords`: Allows Drill to skip malformed records and recover without throwing exceptions.
+* `skipMalformedDocument`: Allows Drill to skip entire malformed documents without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection configuration as shown below:
+
+```json
+
+"jsonOptions": {
+ "allTextMode": true,
+ "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data. Drill allows you to provide a schema for individual endpoints. You can do this in one of three ways:
+
+1. By providing a schema inline [See: Specifying Schema as Table Function Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema. This is an advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+ "providedSchema": [
Review Comment:
You can sort of do that now. The issue I ran into with the `TupleMetadata` representation is that it gets very complicated when you have nested fields. The current implementation allows you to either use the simplified implementation or you can pass a serialized `TupleMetadata` object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre merged pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
cgivre merged PR #2526:
URL: https://github.com/apache/drill/pull/2526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on a diff in pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
cgivre commented on code in PR #2526:
URL: https://github.com/apache/drill/pull/2526#discussion_r864097045
##########
contrib/storage-http/JSON_Options.md:
##########
@@ -0,0 +1,125 @@
+# JSON Options and Configuration
+
+Drill has a collection of JSON configuration options to allow you to configure how Drill interprets JSON files. These are set at the global level, however the HTTP plugin
+allows you to configure these options individually per connection and override the Drill defaults. The options are:
+
+* `allowNanInf`: Configures the connection to interpret `NaN` and `Inf` values
+* `allTextMode`: By default, Drill attempts to infer data types from JSON data. If the data is malformed, Drill may throw schema change exceptions. If your data is
+ inconsistent, you can enable `allTextMode` which when true, Drill will read all JSON values as strings, rather than try to infer the data type.
+* `readNumbersAsDouble`: By default Drill will attempt to interpret integers, floating point number types and strings. One challenge is when data is consistent, Drill may
+ throw schema change exceptions. In addition to `allTextMode`, you can make Drill less sensitive by setting the `readNumbersAsDouble` to `true` which causes Drill to read all
+ numeric fields in JSON data as `double` data type rather than trying to distinguish between ints and doubles.
+* `enableEscapeAnyChar`: Allows a user to escape any character with a \
+* `skipMalformedRecords`: Allows Drill to skip malformed records and recover without throwing exceptions.
+* `skipMalformedDocument`: Allows Drill to skip entire malformed documents without throwing errors.
+
+All of these can be set by adding the `jsonOptions` to your connection configuration as shown below:
+
+```json
+
+"jsonOptions": {
+ "allTextMode": true,
+ "readNumbersAsDouble": true
+}
+
+```
+
+## Schema Provisioning
+One of the challenges of querying APIs is inconsistent data. Drill allows you to provide a schema for individual endpoints. You can do this in one of three ways:
+
+1. By providing a schema inline [See: Specifying Schema as Table Function Parameter](https://drill.apache.org/docs/plugin-configuration-basics/#specifying-the-schema-as-table-function-parameter)
+2. By providing a schema in the configuration for the endpoint.
+3. By providing a serialized TupleMetadata of the desired schema. This is an advanced functionality and should only be used by advanced Drill users.
+
+The schema provisioning currently supports complex types of Arrays and Maps at any nesting level.
+
+### Example Schema Provisioning:
+```json
+"jsonOptions": {
+ "providedSchema": [
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
cgivre commented on PR #2526:
URL: https://github.com/apache/drill/pull/2526#issuecomment-1115032166
>
@vvysotskyi Thanks for the review. I added the logic to the `HttpBatchReader` to support inline schema, however in doing so, I seem to have a found a bug with the `SchemaNegotiator` in that the schema always seems to be `null` with the inline schema.
I created [DRILL-8205](https://issues.apache.org/jira/browse/DRILL-8205) to address this. I left the logic and a unit test so once we have a fix for DRILL-8205 it should all work as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] vvysotskyi commented on pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on PR #2526:
URL: https://github.com/apache/drill/pull/2526#issuecomment-1114877640
Metastore with JSON should also work fine, here is the unit test that checks it: `TestMetastoreWithEasyFormatPlugin.testAnalyzeOnJsonTable`.
For now, drill metastore supports only easy file formats and parquet, but in the future, it could handle the HTTP plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] jnturton commented on pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
jnturton commented on PR #2526:
URL: https://github.com/apache/drill/pull/2526#issuecomment-1114855231
This comment is similar to the last one from @vvysotskyi except that I was experimenting with the metastore. While I cannot see that `CREATE SCHEMA` could work for the http storage plugin, I think it is still possible in theory to store a schema in the metastore something like the following.
```
ANALYZE TABLE table(http.tpch_test_svc.`/nation.json` (type=>'json',
schema=>'inline=(
`n_nationkey` INT not null,
`n_name` VARCHAR not null,
`n_regionkey` DOUBLE not null,
`n_comment` VARCHAR not null)'
)) REFRESH METADATA;
```
Note that the above command works for local CSV but I got a Calcite error for a local JSON file. I did not test it with the HTTP plugin.
Some information about provided schema priority, should it be of interest: https://drill.apache.org/docs/using-drill-metastore/#schema-priority.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] jnturton commented on pull request #2526: DRILL-8204: Allow Provided Schema for HTTP Plugin in JSON Mode
Posted by GitBox <gi...@apache.org>.
jnturton commented on PR #2526:
URL: https://github.com/apache/drill/pull/2526#issuecomment-1114915565
> For now, drill metastore supports only easy file formats and parquet, but in the future, it could handle the HTTP plugin.
@vvysotskyi I think that the HTTP plugin makes use of the same readers as the easy format plugins (CSV, JSON, XML)? Does that mean that metastore might already work with HTTP, or are there likely to be pieces missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org