You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@graphar.apache.org by "SemyonSinchenko (via GitHub)" <gi...@apache.org> on 2025/06/19 16:00:58 UTC
[I] feat(format): format does not specify an element type for list [incubator-graphar]
SemyonSinchenko opened a new issue, #707:
URL: https://github.com/apache/incubator-graphar/issues/707
### Describe the enhancement requested
While LIST itself is specified in format spec (protobuf), the element type does not specified anyhow and serialization the protobuf will result in just "list". But in C++ implementation it is `list<element_type>`, for example, `list<float>`.
### Component(s)
Format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3140956930
> Hi @SemyonSinchenko @yangxk1.
>
> Maybe there is a solution with `google.protobuf.Any`, but I agree with @SemyonSinchenko that:
>
> > the idea failed and we should just drop these protos
>
> Protos make many things heavy, what we need is a utilized format in different SDK in schema leavle, rather than trans many files like JSON.
>
> I agree that we should have an official discussion about deprecating protos, and how to deal with the next step.
Let me start an official discussion in a mailing list about removing these protos until it is not so late and easy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3139249373
We can start an official discussion on a mailing list about this issue , what do you think @Thespica ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3044652073
> @SemyonSinchenko Hi! `List` may look like `List<Object>` in java, but isn't ideal in terms of type safety and clarity. Maybe we can refer to the design of `com.sun.org.apache.bcel.internal.generic.Type` in Spark.
But why we do not want to specify the element type in format spec? At the moment even cpp implementation does not follow the spec because it specifies the element type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "unical1988 (via GitHub)" <gi...@apache.org>.
unical1988 commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3139377428
Why cant we sync through periodic (monthly) gmeet/huddles?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3160788774
@yangxk1 @Thespica @unical1988 I opened a discussion: https://lists.apache.org/list.html?dev@graphar.apache.org
In the case there won't be any vote against it, I will open a voting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "unical1988 (via GitHub)" <gi...@apache.org>.
unical1988 commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3160796067
@SemyonSinchenko @yangxk1 @Thespica I am ok with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3138475312
Hi @SemyonSinchenko ,
I've been thinking more about these enums, and I think they represent `type.id`, they only carry the basic type identifier without nesting information. The nested structure is represented in the `children` field. (e.g. `Type{id=DataType::LIST, children=Type{id=DataType::BOOL, children=NULL}}` representes `List<Bool>`).
I'm curious what is the intended purpose of using protobuf here? Is to standardize how different language bindings parse type information from the `YAML` schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3138507115
> Hi @SemyonSinchenko ,
>
> I've been thinking more about these enums, and I think they represent `type.id`, they only carry the basic type identifier without nesting information. The nested structure is represented in the `children` field. (e.g. `Type{id=DataType::LIST, children=Type{id=DataType::BOOL, children=NULL}}` representes `List<Bool>`).
> I'm curious what is the intended purpose of using protobuf here? Is to standardize how different language bindings parse type information from the `YAML` schema?
The initial idea of using protobuf was to provide a way to work with stabdard ser-de. In theory, protobuf can be converted to JSON and most tools can convert JSON to YAML. And back too. Like different languages should not maintain models of metadata files internally but just re-use these proto that can be published as a package.
But further we are going, Im thinking that more and more the idea failed and we should just drop these protos. For example, the whole new Java API re-creates all the models on top of these protos anyway and protos doesnt simplify the story but add complexity.
cc: @acezen @Thespica
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3138977098
I would really like to discuss this topic, especially before the whole Java API will built on top of these proto and the cost of refactoring will be very high. Can we have a community call about it? Alternatively I can start an official discussion on a mailing list to follow ASF rules. @yangxk1 what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3003345490
@Thespica Hi! What do you think about it? As I remember, at the moment proto-specs are used inside Java only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-2988590769
It cannot be [just a enum](https://github.com/apache/incubator-graphar/blob/main/format/enums.proto#L34), except the case we want to have something like this:
```protobuf
enum DataType {
BOOL = 0;
INT32 = 1;
INT64 = 2;
FLOAT = 3;
DOUBLE = 4;
STRING = 5;
DATE = 6;
TIMESTAMP = 7;
TIME = 8;
LIST_BOOL = 9;
LIST_INT32 = 10;
LIST_INT64 = 11;
...
};
```
but in my understanding it will be impossible to maintain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3044661740
Here (https://github.com/apache/incubator-graphar-testing/blob/main/ldbc_sample%2Fparquet%2Fperson_with_feature.vertex.yml) it is a "typed" list, not just list. Should we specify it in format spec?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3044641369
@SemyonSinchenko Hi! `List` may look like `List<Object>` in java, but isn't ideal in terms of type safety and clarity. Maybe we can refer to the design of `com.sun.org.apache.bcel.internal.generic.Type` in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "unical1988 (via GitHub)" <gi...@apache.org>.
unical1988 commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3002131099
Could i take this to work on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-2988597497
My proposal is to make each possible data type as a message, like this:
```protobuf
message Bool {
string name = 1;
}
message Int32 {
string name = 1;
}
...
message List {
string name = 1;
onof element_type {
Bool bool_type = 2;
Int32 int32_type = 3;
...
}
}
```
but it is a breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "yangxk1 (via GitHub)" <gi...@apache.org>.
yangxk1 commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3044759151
You're right.
The current implementation in Java cannot handle `List<float>`
The C++ implementation is similar to your first proposal.
https://github.com/apache/incubator-graphar/blob/712b204df2aa625d233302a27bc83bd8affd2229/cpp/src/graphar/types.cc#L119
we can use it If we don’t need to support nested types (e.g., `List<List<float>>`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org
Re: [I] feat(format): format does not specify an element type for list [incubator-graphar]
Posted by "Thespica (via GitHub)" <gi...@apache.org>.
Thespica commented on issue #707:
URL: https://github.com/apache/incubator-graphar/issues/707#issuecomment-3139365961
Hi @SemyonSinchenko @yangxk1.
Although maybe there is a solution with `google.protobuf.Any`, but I agree with @SemyonSinchenko that:
> the idea failed and we should just drop these protos
Protos make many things heavy, what we need is a utilized format in different SDK, rather than trans many files like JSON.
I agree that we should have an official discussion about deprecating protos, and how to deal with the next step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@graphar.apache.org
For additional commands, e-mail: commits-help@graphar.apache.org