You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/05/21 09:14:04 UTC
[GitHub] [arrow] Jokser opened a new issue, #13208: Protobuf specification for arrow types
Jokser opened a new issue, #13208:
URL: https://github.com/apache/arrow/issues/13208
Hi everybody.
I develop gRPC services that operate with arrow types in order to perform some DDL actions (e.g. create a table with columns that have arrow types).
In order to do it, I have to repeat arrow type enums and other things related to arrow type configuration (scale, precision for decimal, time units for timestamp/time types, dictionary encoding, etc.)
Example:
```
// 1-1 mapping to arrow::TimeUnit.
enum TimeUnit {
NONE = 0; // Used as indicator that unit is not set.
SECONDS = 1;
MILLISECONDS = 2;
MICROSECONDS = 3;
NANOSECONDS = 4;
}
enum ArrowType {
NA = 0; // Used as indicator that type is not set
BOOL = 1;
UINT8 = 2;
INT8 = 3;
UINT16 = 4;
INT16 = 5;
UINT32 = 6;
INT32 = 7;
UINT64 = 8;
INT64 = 9;
HALF_FLOAT = 10;
FLOAT = 11;
DOUBLE = 12;
STRING = 13;
BINARY = 14;
FIXED_SIZE_BINARY = 15;
DATE32 = 16;
DATE64 = 17;
TIMESTAMP = 18;
TIME32 = 19;
TIME64 = 20;
INTERVAL_MONTHS = 21;
INTERVAL_DAY_TIME = 22;
DECIMAL = 23;
DECIMAL256 = 24;
LIST = 25;
STRUCT = 26;
SPARSE_UNION = 27;
DENSE_UNION = 28;
DICTIONARY = 29;
MAP = 30;
EXTENSION = 31;
FIXED_SIZE_LIST = 32;
DURATION = 33;
LARGE_STRING = 34;
LARGE_BINARY = 35;
LARGE_LIST = 36;
INTERVAL_MONTH_DAY_NANO = 37;
}
/// And so on.
```
On the server-side, I do enum casts and manual arrow types constructions in order to get complete arrow::Type for a configured column.
It would be much easier to have a ready protobuf specification for that thing and a plugin that will generate protobuf classes with an implicit conversion to arrow::Type.
WDYT about it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Jokser commented on issue #13208: Protobuf specification for arrow types
Posted by GitBox <gi...@apache.org>.
Jokser commented on issue #13208:
URL: https://github.com/apache/arrow/issues/13208#issuecomment-1133649363
@lidavidm Thank you. I'll look in this direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #13208: Protobuf specification for arrow types
Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #13208:
URL: https://github.com/apache/arrow/issues/13208#issuecomment-1134548212
I'll close this now but feel free to re-open if you have questions about that approach. We can also add examples of the serialization/deserialization part to the Arrow Cookbook since I don't think it's necessarily obvious in all languages (especially Java).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed issue #13208: Protobuf specification for arrow types
Posted by GitBox <gi...@apache.org>.
lidavidm closed issue #13208: Protobuf specification for arrow types
URL: https://github.com/apache/arrow/issues/13208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #13208: Protobuf specification for arrow types
Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #13208:
URL: https://github.com/apache/arrow/issues/13208#issuecomment-1133604025
Arrow Flight just encodes Arrow schemas by serializing/deserializing them and using a bytes field. Would that work? That way you wouldn't need to re-model schemas in Protobuf and you could use things like field/schema metadata as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org