You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Michael Dürig <mi...@gmail.com> on 2012/04/12 19:09:02 UTC
Encoding of JCR values in json
Hi,
We need to come up with an encoding scheme for JCR values in JSON. While
String and Boolean are straight forward double, long and decimal are
already more troublesome. Finally for binary, date, name, path,
reference, weakreference and uri there is no direct correspondence in JSON.
The way I solved this in spi2microkernel [1] is by encoding values by
serializing them to their string representation (Value.getString()) and
prepend its property type (value.getType) in radix 16 and a colon (:).
So "A:123" would stand for a decimal value of 123. Boolean, long and
some strings don't need the "x:" pre-amble since they can be
unambiguously encoded to the respective JSON representation. Only String
where the second character is a colon need the "x:" preamble for
disambiguation:
"example":{
"long":123,
"another long":"3:124",
"double":"4:123.4",
"string":"foo",
"another string":"1:a:string"
}
WDYT? Other suggestions?
On a related note: what kind of values do we want to expose from
oak-core? JSON like or JCR like? Implementation wise, would that
en/decoding happen inside oak-jcr or oak-core?
Michael
[1]
http://svn.apache.org/viewvc/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/Values.java?view=markup
Re: Encoding of JCR values in json
Posted by Julian Reschke <ju...@gmx.de>.
On 2012-04-13 13:43, Jukka Zitting wrote:
> Hi,
>
> On Thu, Apr 12, 2012 at 11:46 PM, Michael Dürig<md...@apache.org> wrote:
>>> [value.equals(Long.valueOf(value).toString())]
>>
>> That's the approach I used in spi2microkernel. However, it has the drawback
>> that we need to try catch through the cases when using the valueOf methods
>> for determining the exact numeric type. Or we write our own code for
>> parsing...
>
> We need a JSON parser in any case, so that's probably the best place
> to take care of this.
>
>>> [infinities and NaN values]
>>
>> Ack. Either way is fine with me.
>
> I suggest we choose to not support such values for now. If it later
> turns out that support for them is needed, it should be fairly
> straightforward to implement using the typing mechanism we in any case
> need.
>
>>> * The explicit encoding of numeric constants from JCR seems a bit
>>> troublesome and makes potential extensions more cumbersome.
>>
>> What extensions come to mind?
>
> For example some applications might well want to be able to store
> things like coordinates (think of geospatial search) as individual
> scalars instead of collections of properties. Or there might be a
> desire to have a custom property type for say email addresses or phone
> numbers just like we have one for URLs in JCR 2.0. Or there could be a
> need for storing date and time values separately instead of the
> current timestamp values used in JCR. We don't have a current need for
> things like these, but a future JCR version or another standard or use
> case we want to support might well change that. Having an extensible
> typing mechanism makes implementing such things easy.
>
>>> * The overloading of normal strings requires that all string values
>>> will need to be checked for whether they need to be escaped.
>>
>> That's a small penalty:
>>
>> if(s.length()>= 2&& s.charAt(1) == ':') { ... }
>
> Yep, it's not too big a deal, though there is a conceptual step from
> "no escaping" to "some escaping".
>
>> The trouble with that is, that type info and value are spread across
>> different properties. Setting a JCR property requires two JSON diff
>> operations here.
>
> I don't think that would be too troublesome since the standard string
> and number formats cover the majority of properties.
>
>> Worse for JCR observation: the corresponding set property entries
>> might be spread across the journal.
>
> I'm not sure I follow. You mean a case where the value is set in one
> revision and the type in another? That should be interpreted as two
> setProperty() operations, each with different value types.
>
>> So there are three different approaches now: 1) encoding the type into the
>> value, 2) encoding it into a separate property, or 3) encoding it into the
>> name of the property.
>
> Microsoft chose a fairly smart variant of 1 for ASP.NET, see
> http://weblogs.asp.net/bleroy/archive/2008/01/18/dates-and-json.aspx
Outch.
> For us that would mean something like this:
>
> "example":{
> "long":123,
> "another long":"\/Long(124)",
> "double":"\/Double(123.4)",
> "string":"foo",
> "another string":"/string"
> }
>
> The only trouble is that for this to work we need to require the
> MicroKernel to keep track of the exact character sequences instead of
> just the logical, unescaped strings. Currently this indeed is the
> case.
But in that case out datamodel isn't JSON anymore. Will it work with
MongoDB?
The good thing about JSON is that it's simple. The bad thing is that
it's sometimes too simple.
Best regards, Julian
Re: Encoding of JCR values in json
Posted by Julian Reschke <ju...@gmx.de>.
On 2012-04-13 14:24, Michael Dürig wrote:
>
>
> On 13.4.12 12:43, Jukka Zitting wrote:
>
>>>> * The explicit encoding of numeric constants from JCR seems a bit
>>>> troublesome and makes potential extensions more cumbersome.
>>>
>>> What extensions come to mind?
>>
>> For example some applications might well want to be able to store
>> things like coordinates (think of geospatial search) as individual
>> scalars instead of collections of properties. Or there might be a
>> desire to have a custom property type for say email addresses or phone
>> numbers just like we have one for URLs in JCR 2.0. Or there could be a
>> need for storing date and time values separately instead of the
>> current timestamp values used in JCR. We don't have a current need for
>> things like these, but a future JCR version or another standard or use
>> case we want to support might well change that. Having an extensible
>> typing mechanism makes implementing such things easy.
>
> Makes sense. If we get rid of the explicit encoding using the numerical
> constants from PropertyType and use a more open approach we should be
> fine extension wise however.
>
> [...]
>
>>> The trouble with that is, that type info and value are spread across
>>> different properties. Setting a JCR property requires two JSON diff
>>> operations here.
>>
>> I don't think that would be too troublesome since the standard string
>> and number formats cover the majority of properties.
>>
>>> Worse for JCR observation: the corresponding set property entries
>>> might be spread across the journal.
>>
>> I'm not sure I follow. You mean a case where the value is set in one
>> revision and the type in another? That should be interpreted as two
>> setProperty() operations, each with different value types.
>
> What I meant is: creating the JCR observation events from the
> Microkernel journal might be troublesome since the two setProperty
> operations (one for the value and the other for the type) are not
> guaranteed to be anywhere close to each other in the journal.
>
>>
>>> So there are three different approaches now: 1) encoding the type
>>> into the
>>> value, 2) encoding it into a separate property, or 3) encoding it
>>> into the
>>> name of the property.
>>
>> Microsoft chose a fairly smart variant of 1 for ASP.NET, see
>> http://weblogs.asp.net/bleroy/archive/2008/01/18/dates-and-json.aspx
>>
>> For us that would mean something like this:
>>
>> "example":{
>> "long":123,
>> "another long":"\/Long(124)",
>> "double":"\/Double(123.4)",
>> "string":"foo",
>> "another string":"/string"
>> }
>>
>> The only trouble is that for this to work we need to require the
>> MicroKernel to keep track of the exact character sequences instead of
>> just the logical, unescaped strings. Currently this indeed is the
>> case.
>
> Tom brought that up earlier already. I'm a bit sceptic about using a
> loop hole in the spec since clients using standard JSON parsers might
> not be able to distinguish between the long value
>
> "a long":"\/Long(124)"
>
> and the string value
>
> "a string":"/Long(124)"
>
> because in both cases they only see the un-escaped value which is the same.
Absolutely.
I wouldn't call it a "loophole". The trick just relies on an
implementation detail of a specific JSON implementation. (Which reminds
me of member ordering :-).
> ...
Best regards, Julian
Re: Encoding of JCR values in json
Posted by Michael Dürig <md...@apache.org>.
On 13.4.12 12:43, Jukka Zitting wrote:
>>> * The explicit encoding of numeric constants from JCR seems a bit
>>> troublesome and makes potential extensions more cumbersome.
>>
>> What extensions come to mind?
>
> For example some applications might well want to be able to store
> things like coordinates (think of geospatial search) as individual
> scalars instead of collections of properties. Or there might be a
> desire to have a custom property type for say email addresses or phone
> numbers just like we have one for URLs in JCR 2.0. Or there could be a
> need for storing date and time values separately instead of the
> current timestamp values used in JCR. We don't have a current need for
> things like these, but a future JCR version or another standard or use
> case we want to support might well change that. Having an extensible
> typing mechanism makes implementing such things easy.
Makes sense. If we get rid of the explicit encoding using the numerical
constants from PropertyType and use a more open approach we should be
fine extension wise however.
[...]
>> The trouble with that is, that type info and value are spread across
>> different properties. Setting a JCR property requires two JSON diff
>> operations here.
>
> I don't think that would be too troublesome since the standard string
> and number formats cover the majority of properties.
>
>> Worse for JCR observation: the corresponding set property entries
>> might be spread across the journal.
>
> I'm not sure I follow. You mean a case where the value is set in one
> revision and the type in another? That should be interpreted as two
> setProperty() operations, each with different value types.
What I meant is: creating the JCR observation events from the
Microkernel journal might be troublesome since the two setProperty
operations (one for the value and the other for the type) are not
guaranteed to be anywhere close to each other in the journal.
>
>> So there are three different approaches now: 1) encoding the type into the
>> value, 2) encoding it into a separate property, or 3) encoding it into the
>> name of the property.
>
> Microsoft chose a fairly smart variant of 1 for ASP.NET, see
> http://weblogs.asp.net/bleroy/archive/2008/01/18/dates-and-json.aspx
>
> For us that would mean something like this:
>
> "example":{
> "long":123,
> "another long":"\/Long(124)",
> "double":"\/Double(123.4)",
> "string":"foo",
> "another string":"/string"
> }
>
> The only trouble is that for this to work we need to require the
> MicroKernel to keep track of the exact character sequences instead of
> just the logical, unescaped strings. Currently this indeed is the
> case.
Tom brought that up earlier already. I'm a bit sceptic about using a
loop hole in the spec since clients using standard JSON parsers might
not be able to distinguish between the long value
"a long":"\/Long(124)"
and the string value
"a string":"/Long(124)"
because in both cases they only see the un-escaped value which is the same.
Nevertheless the more I think about it, the more I'm convinced that the
type should go together with the value. One reason is that the names are
much more often accessed than the values and that the type mostly
matters when accessing the value, not the name.
What about using three character prefixes to denote the type? That
should give us plenty of type designators while parsing would still be
very easy and fast.
"example":{
"long":123,
"another long":"lng:124",
"double":"dbl:123.4",
"string":"foo",
"another string":"str:xyz:string"
}
The type designators could then be registered/reserved somewhere (e.g.
in the system workspace):
designator: lng
jcr-property-type: Long
jcr-property-type-code: 3
Michael
>
> BR,
>
> Jukka Zitting
Re: Encoding of JCR values in json
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Thu, Apr 12, 2012 at 11:46 PM, Michael Dürig <md...@apache.org> wrote:
>> [value.equals(Long.valueOf(value).toString())]
>
> That's the approach I used in spi2microkernel. However, it has the drawback
> that we need to try catch through the cases when using the valueOf methods
> for determining the exact numeric type. Or we write our own code for
> parsing...
We need a JSON parser in any case, so that's probably the best place
to take care of this.
>> [infinities and NaN values]
>
> Ack. Either way is fine with me.
I suggest we choose to not support such values for now. If it later
turns out that support for them is needed, it should be fairly
straightforward to implement using the typing mechanism we in any case
need.
>> * The explicit encoding of numeric constants from JCR seems a bit
>> troublesome and makes potential extensions more cumbersome.
>
> What extensions come to mind?
For example some applications might well want to be able to store
things like coordinates (think of geospatial search) as individual
scalars instead of collections of properties. Or there might be a
desire to have a custom property type for say email addresses or phone
numbers just like we have one for URLs in JCR 2.0. Or there could be a
need for storing date and time values separately instead of the
current timestamp values used in JCR. We don't have a current need for
things like these, but a future JCR version or another standard or use
case we want to support might well change that. Having an extensible
typing mechanism makes implementing such things easy.
>> * The overloading of normal strings requires that all string values
>> will need to be checked for whether they need to be escaped.
>
> That's a small penalty:
>
> if(s.length() >= 2 && s.charAt(1) == ':') { ... }
Yep, it's not too big a deal, though there is a conceptual step from
"no escaping" to "some escaping".
> The trouble with that is, that type info and value are spread across
> different properties. Setting a JCR property requires two JSON diff
> operations here.
I don't think that would be too troublesome since the standard string
and number formats cover the majority of properties.
> Worse for JCR observation: the corresponding set property entries
> might be spread across the journal.
I'm not sure I follow. You mean a case where the value is set in one
revision and the type in another? That should be interpreted as two
setProperty() operations, each with different value types.
> So there are three different approaches now: 1) encoding the type into the
> value, 2) encoding it into a separate property, or 3) encoding it into the
> name of the property.
Microsoft chose a fairly smart variant of 1 for ASP.NET, see
http://weblogs.asp.net/bleroy/archive/2008/01/18/dates-and-json.aspx
For us that would mean something like this:
"example":{
"long":123,
"another long":"\/Long(124)",
"double":"\/Double(123.4)",
"string":"foo",
"another string":"/string"
}
The only trouble is that for this to work we need to require the
MicroKernel to keep track of the exact character sequences instead of
just the logical, unescaped strings. Currently this indeed is the
case.
BR,
Jukka Zitting
Re: Encoding of JCR values in json
Posted by Julian Reschke <ju...@gmx.de>.
On 2012-04-12 23:46, Michael Dürig wrote:
> ...
>> I'd ideally like to keep it JSON-like so we can easily implement a
>> JavaScript-friendly HTTP mapping directly based on the Oak API without
>> having to go through extra levels of mapping.
>
> Hmmm, seems reasonable. What about Angela's concerns?
>
> Michael
> ...
That means that the JCR<->JSON mapping we define here effectively would
become the mapping used in JSOP (if we ever get there), so it would be
part of a public protocol contract.
Not saying that this is a bad idea, but we need to be aware of the
consequences :-)
Best regards, Julian
Re: Encoding of JCR values in json
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Fri, Apr 13, 2012 at 5:25 PM, Michael Dürig <md...@apache.org> wrote:
> On 13.4.12 14:40, Thomas Mueller wrote:
>> We should document this in the MicroKernel specification. The JSON
>> specification itself doesn't define such details.
>
> And this should be done rather sooner than later.
See http://markmail.org/message/vrhizcykjhbfisjg for some earlier
discussion on this, including relevant questions to be answered.
BR,
Jukka Zitting
Re: Encoding of JCR values in json
Posted by Michael Dürig <md...@apache.org>.
On 13.4.12 14:40, Thomas Mueller wrote:
> Hi,
>
>>> An alternative might be: numbers with an "e" are double ("10e0"), numbers
>>> with a dot are decimal ("10.0"), all other numbers are long ("10"). But
>>> that would require the MicroKernel stores the *exact* JSON
>>> representation.
>>> I don't think that's a good idea either.
>>
>> The underlying storage mechanism must not be of concern to the user of
>> the Microkernel API. The Microkernel API uses JSON to transport values.
>> So the Microkernel implementation is in charge of serializing these into
>> a format suitable for the underlying storage mechanism. Even if I put a
>> number with 10 Million digits into a JSON property.
>
> I think we should not rely on the MicroKernel implementation to support
> unlimited precision numbers. For example MongoDB only supports IEEE 754
> floating point numbers, as defined by the BSON spec:
> http://en.wikipedia.org/wiki/BSON - so we can't store BigDecimal as a JSON
> number, because we would risk losing precision.
>
> Also, I would not rely on the MicroKernel implementation to preserve the
> exact JSON representation. If we store the number as 100.00, we should not
> rely on getting 100.00 back. We might as well get 100.0 or 100 back.
>
> We should document this in the MicroKernel specification. The JSON
> specification itself doesn't define such details.
And this should be done rather sooner than later.
>
>> How could the GC possible know whether a
>> binary is still in use or not?
>> String blobId = mk.write(inStream);
>>
>> and write the returned blobId on a piece of paper. According to the
>> current Microkernel contract I could come back after a couple of years
>> and would still be able to retrieve that blob.
>
> Yes, we need to narrow the contract.
Again, we'd rather be quick here.
Michael
>
> Regards,
> Thomas
>
>
Re: Encoding of JCR values in json
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
>>An alternative might be: numbers with an "e" are double ("10e0"), numbers
>> with a dot are decimal ("10.0"), all other numbers are long ("10"). But
>> that would require the MicroKernel stores the *exact* JSON
>>representation.
>> I don't think that's a good idea either.
>
>The underlying storage mechanism must not be of concern to the user of
>the Microkernel API. The Microkernel API uses JSON to transport values.
>So the Microkernel implementation is in charge of serializing these into
>a format suitable for the underlying storage mechanism. Even if I put a
>number with 10 Million digits into a JSON property.
I think we should not rely on the MicroKernel implementation to support
unlimited precision numbers. For example MongoDB only supports IEEE 754
floating point numbers, as defined by the BSON spec:
http://en.wikipedia.org/wiki/BSON - so we can't store BigDecimal as a JSON
number, because we would risk losing precision.
Also, I would not rely on the MicroKernel implementation to preserve the
exact JSON representation. If we store the number as 100.00, we should not
rely on getting 100.00 back. We might as well get 100.0 or 100 back.
We should document this in the MicroKernel specification. The JSON
specification itself doesn't define such details.
>How could the GC possible know whether a
>binary is still in use or not?
>String blobId = mk.write(inStream);
>
>and write the returned blobId on a piece of paper. According to the
>current Microkernel contract I could come back after a couple of years
>and would still be able to retrieve that blob.
Yes, we need to narrow the contract.
Regards,
Thomas
Re: Encoding of JCR values in json
Posted by Michael Dürig <md...@apache.org>.
On 13.4.12 7:00, Thomas Mueller wrote:
> Hi,
>
>>> * A JSON value is a double if
>>> value.equals(Double.valueOf(value).toString())
>>> * A JSON value is a long if value.equals(Long.valueOf(value).toString())
>>> * A JSON value is a decimal if it's a JSON number that matches neither
>>> of the above two rules
>>
>> That's the approach I used in spi2microkernel. However, it has the
>> drawback that we need to try catch through the cases when using the
>> valueOf methods for determining the exact numeric type. Or we write our
>> own code for parsing...
>
> Java BigDecimal values can be very large. I'm not sure if, for example,
> MongoDB has some kind of limitation on numbers (precision, length). Then,
> how would you distinguish between BigDecimal 10, Double 10, and Long 10?
> An alternative might be: numbers with an "e" are double ("10e0"), numbers
> with a dot are decimal ("10.0"), all other numbers are long ("10"). But
> that would require the MicroKernel stores the *exact* JSON representation.
> I don't think that's a good idea either.
The underlying storage mechanism must not be of concern to the user of
the Microkernel API. The Microkernel API uses JSON to transport values.
So the Microkernel implementation is in charge of serializing these into
a format suitable for the underlying storage mechanism. Even if I put a
number with 10 Million digits into a JSON property.
If we want/need to make restrictions here, we will have to clarify and
document these. See OAK-11.
>
> So I would prefer explicit typing, except for Long.
>
>>> I'd be OK with us explicitly *not* supporting special cases like
>>> infinities and NaN values.
>
> I would prefer supporting them, using a well defined syntax (as a String).
>
>>> Right. There's an additional constraint for binary values in that the
>>> MicroKernel garbage collector needs some way to connect JSON
>>> properties to referenced binaries. It would be useful if the same
>>> convention was used also higher up the stack.
>>
>> Good point. Maybe Dominique can provide some insight here?
>
> There are two garbage collectors in the MK: the "node data" GC and the
> "data store" GC. I guess this is about the data store GC, which I wrote,
> not Dominique.
>
> Currently the data store GC is on a high level. Marking binaries that are
> still in use is done using MicroKernel.getLength(String blobId). But data
> store GC needs to traverse all nodes in all available revisions, so it is
> really slow. It would be nice if binaries can be indexed, so that garbage
> collection doesn't have to traverse *all* nodes in the repository (in all
> revisions). So binary references should be easy to recognize.
Not sure whether I understand. How could the GC possible know whether a
binary is still in use or not? I could do
String blobId = mk.write(inStream);
and write the returned blobId on a piece of paper. According to the
current Microkernel contract I could come back after a couple of years
and would still be able to retrieve that blob.
Michael
>
>> The way I solved this in spi2microkernel [1] is by encoding values by
>> serializing them to their string representation (Value.getString()) and
>> prepend its property type (value.getType) in radix 16 and a colon (:).
>
> I think it's a good solution. It has been proven to be robust so far. Even
> thought, for debugging purposes, I would prefer not to use hex digits. But
> that's a minor issue really.
>
>>>> On a related note: what kind of values do we want to expose from
>>>> oak-core?
>>>> JSON like or JCR like?
>
> I would use JCR like. The (oak-jcr to oak-core) remoting implementation
> might use the same JSON conversion as used between oak-core and oak-mk,
> but do we really need to define this now?
>
> Regards,
> Thomas
>
Re: Encoding of JCR values in json
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
>>* A JSON value is a double if
>>value.equals(Double.valueOf(value).toString())
>> * A JSON value is a long if value.equals(Long.valueOf(value).toString())
>> * A JSON value is a decimal if it's a JSON number that matches neither
>> of the above two rules
>
>That's the approach I used in spi2microkernel. However, it has the
>drawback that we need to try catch through the cases when using the
>valueOf methods for determining the exact numeric type. Or we write our
>own code for parsing...
Java BigDecimal values can be very large. I'm not sure if, for example,
MongoDB has some kind of limitation on numbers (precision, length). Then,
how would you distinguish between BigDecimal 10, Double 10, and Long 10?
An alternative might be: numbers with an "e" are double ("10e0"), numbers
with a dot are decimal ("10.0"), all other numbers are long ("10"). But
that would require the MicroKernel stores the *exact* JSON representation.
I don't think that's a good idea either.
So I would prefer explicit typing, except for Long.
>>I'd be OK with us explicitly *not* supporting special cases like
>> infinities and NaN values.
I would prefer supporting them, using a well defined syntax (as a String).
>>Right. There's an additional constraint for binary values in that the
>> MicroKernel garbage collector needs some way to connect JSON
>> properties to referenced binaries. It would be useful if the same
>> convention was used also higher up the stack.
>
>Good point. Maybe Dominique can provide some insight here?
There are two garbage collectors in the MK: the "node data" GC and the
"data store" GC. I guess this is about the data store GC, which I wrote,
not Dominique.
Currently the data store GC is on a high level. Marking binaries that are
still in use is done using MicroKernel.getLength(String blobId). But data
store GC needs to traverse all nodes in all available revisions, so it is
really slow. It would be nice if binaries can be indexed, so that garbage
collection doesn't have to traverse *all* nodes in the repository (in all
revisions). So binary references should be easy to recognize.
>The way I solved this in spi2microkernel [1] is by encoding values by
> serializing them to their string representation (Value.getString()) and
> prepend its property type (value.getType) in radix 16 and a colon (:).
I think it's a good solution. It has been proven to be robust so far. Even
thought, for debugging purposes, I would prefer not to use hex digits. But
that's a minor issue really.
>>>On a related note: what kind of values do we want to expose from
>>>oak-core?
>>> JSON like or JCR like?
I would use JCR like. The (oak-jcr to oak-core) remoting implementation
might use the same JSON conversion as used between oak-core and oak-mk,
but do we really need to define this now?
Regards,
Thomas
Re: Encoding of JCR values in json
Posted by Michael Dürig <md...@apache.org>.
On 12.4.12 19:15, Jukka Zitting wrote:
>> While String and Boolean are straight forward double, long and decimal
>> are already more troublesome.
>
> As basic rules for handling the latter types I'd define something like this:
>
> * A JSON value is a double if value.equals(Double.valueOf(value).toString())
> * A JSON value is a long if value.equals(Long.valueOf(value).toString())
> * A JSON value is a decimal if it's a JSON number that matches neither
> of the above two rules
That's the approach I used in spi2microkernel. However, it has the
drawback that we need to try catch through the cases when using the
valueOf methods for determining the exact numeric type. Or we write our
own code for parsing...
> I'd be OK with us explicitly *not* supporting special cases like
> infinities and NaN values. We'd just throw ValueFormatExceptions for
> them on the oak-jcr level and IllegalArgumentExceptions on the
> oak-core (or do something similar). Alternatively we should use
> explicit typing information with a well defined syntax for expressing
> such special cases.
Ack. Either way is fine with me.
>
>> Finally for binary, date, name, path, reference, weakreference and uri
>> there is no direct correspondence in JSON.
>
> Right. There's an additional constraint for binary values in that the
> MicroKernel garbage collector needs some way to connect JSON
> properties to referenced binaries. It would be useful if the same
> convention was used also higher up the stack.
Good point. Maybe Dominique can provide some insight here?
>> The way I solved this in spi2microkernel [1] is by encoding values by
>> serializing them to their string representation (Value.getString()) and
>> prepend its property type (value.getType) in radix 16 and a colon (:).
>
> Sounds like a workable solution, though I have some reservations:
>
> * The explicit encoding of numeric constants from JCR seems a bit
> troublesome and makes potential extensions more cumbersome.
What extensions come to mind?
>
> * The overloading of normal strings requires that all string values
> will need to be checked for whether they need to be escaped.
That's a small penalty:
if(s.length() >= 2 && s.charAt(1) == ':') { ... }
>
> An alternative solution would be to use something like the @TypeHint
> feature used by the JSON functionality in Sling. Instead of "@", we
> should use something like "::" that's invalid in a JCR name to prevent
> conflicts. With such a solution the example JSON object would look
> like this:
>
> "example":{
> "long":123,
> "another long":"124",
> "another long::TypeHint":"long",
> "double":"123.4",
> "double::TypeHint":"double",
> "string":"foo",
> "another string":"a:string",
> "another string::TypeHint":"string"
> }
>
> That's a bit verbose, so we could also put the type hint directly into
> the relevant property name, like this:
The trouble with that is, that type info and value are spread across
different properties. Setting a JCR property requires two JSON diff
operations here. Worse for JCR observation: the corresponding set
property entries might be spread across the journal.
>
> "example":{
> "long":123,
> "another long::long":"124",
> "double::double":"123.4",
> "string":"foo",
> "another string::string":"a:string"
> }
>
> The main downsides of this approach are:
>
> * Name-based property accesses will potentially need to traverse
> through all properties to find a matching name. That should be
> manageable since the implementation can pre-scan all property names
> and split them to name and type parts.
>
> * There's a potential for conflicts like when a JSON object contains
> both "x" and "x::long" properties. That can be dealt with in a commit
> validator that prevents such objects from being persisted.
So there are three different approaches now: 1) encoding the type into
the value, 2) encoding it into a separate property, or 3) encoding it
into the name of the property.
I think 2) is most troublesome for the reasons outlined above. Regarding
1) and 3) we should also think about consequences for query and
indexing. Are there any drawbacks, advantages for either of those? Tom?
>
>> On a related note: what kind of values do we want to expose from oak-core?
>> JSON like or JCR like?
>
> I'd ideally like to keep it JSON-like so we can easily implement a
> JavaScript-friendly HTTP mapping directly based on the Oak API without
> having to go through extra levels of mapping.
Hmmm, seems reasonable. What about Angela's concerns?
Michael
>
>> Implementation wise, would that en/decoding happen inside oak-jcr or oak-core?
>
> I'd put the JSON-JCR type mapping into a shared helper class in
> oak-core since it'll be needed by a lot of things like query and node
> type handling inside oak-core. But the API interfaces should IMO be
> based on JSON types to support cases where JCR typing isn't needed or
> wanted.
>
> BR,
>
> Jukka Zitting
Re: Encoding of JCR values in json
Posted by Angela Schreiber <an...@adobe.com>.
hi michael
>> having said that: thinking of a ValueFactory implementation
>> in oak-jcr that creates Value objects that can server both as
>> jcr-values and oak-values. the internal representation could
>> still be anything including json-representation... but we
>> wouldn't be limited to JSON types on the API level.
>
> This sounds like an interesting idea. It kind of covers the middle
> ground between both approaches. Only that I'd put that implementation
> into oak-core, not into oak-jcr.
fine with me.
i didn't dare suggesting this as it would be yet another
reason for having a dependency to javax.jcr in the oak-core
project... (which i don't have too much problem with).
angela
Re: Encoding of JCR values in json
Posted by Michael Dürig <md...@apache.org>.
On 13.4.12 8:12, Angela Schreiber wrote:
> hi
>
>>> On a related note: what kind of values do we want to expose from
>>> oak-core?
>>> JSON like or JCR like?
>>
>> I'd ideally like to keep it JSON-like so we can easily implement a
>> JavaScript-friendly HTTP mapping directly based on the Oak API without
>> having to go through extra levels of mapping.
>>
>>> Implementation wise, would that en/decoding happen inside oak-jcr or
>>> oak-core?
>>
>> I'd put the JSON-JCR type mapping into a shared helper class in
>> oak-core since it'll be needed by a lot of things like query and node
>> type handling inside oak-core. But the API interfaces should IMO be
>> based on JSON types to support cases where JCR typing isn't needed or
>> wanted.
>
> having a mapping to JSON is required anyway. the basic question
> for me is whether oak-api should use that natively. and in which
> cases jcr typing wasn't needed/wanted.
>
> based on my assumption that in our default use-case there is
> no remoting between oak-jcr and oak-core i just see the drawback
> of converting jcr-Value to json (oak-jcr) and parse it again
> to build some sort of oak-Value that have all type information
> for validation... instead of having that abstracted in an oak-Value
> interface.
>
> since we don't use qualified names, the jcr-value in most
> cases was very similar to the oak-value.
>
> having said that: thinking of a ValueFactory implementation
> in oak-jcr that creates Value objects that can server both as
> jcr-values and oak-values. the internal representation could
> still be anything including json-representation... but we
> wouldn't be limited to JSON types on the API level.
This sounds like an interesting idea. It kind of covers the middle
ground between both approaches. Only that I'd put that implementation
into oak-core, not into oak-jcr.
Michael
>
> kind regards
> angela
>
>
>> BR,
>>
>> Jukka Zitting
Re: Encoding of JCR values in json
Posted by Angela Schreiber <an...@adobe.com>.
hi
>> On a related note: what kind of values do we want to expose from oak-core?
>> JSON like or JCR like?
>
> I'd ideally like to keep it JSON-like so we can easily implement a
> JavaScript-friendly HTTP mapping directly based on the Oak API without
> having to go through extra levels of mapping.
>
>> Implementation wise, would that en/decoding happen inside oak-jcr or oak-core?
>
> I'd put the JSON-JCR type mapping into a shared helper class in
> oak-core since it'll be needed by a lot of things like query and node
> type handling inside oak-core. But the API interfaces should IMO be
> based on JSON types to support cases where JCR typing isn't needed or
> wanted.
having a mapping to JSON is required anyway. the basic question
for me is whether oak-api should use that natively. and in which
cases jcr typing wasn't needed/wanted.
based on my assumption that in our default use-case there is
no remoting between oak-jcr and oak-core i just see the drawback
of converting jcr-Value to json (oak-jcr) and parse it again
to build some sort of oak-Value that have all type information
for validation... instead of having that abstracted in an oak-Value
interface.
since we don't use qualified names, the jcr-value in most
cases was very similar to the oak-value.
having said that: thinking of a ValueFactory implementation
in oak-jcr that creates Value objects that can server both as
jcr-values and oak-values. the internal representation could
still be anything including json-representation... but we
wouldn't be limited to JSON types on the API level.
kind regards
angela
> BR,
>
> Jukka Zitting
Re: Encoding of JCR values in json
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Thu, Apr 12, 2012 at 7:09 PM, Michael Dürig <mi...@gmail.com> wrote:
> We need to come up with an encoding scheme for JCR values in JSON.
Indeed. Thanks for bringing this up!
> While String and Boolean are straight forward double, long and decimal
> are already more troublesome.
As basic rules for handling the latter types I'd define something like this:
* A JSON value is a double if value.equals(Double.valueOf(value).toString())
* A JSON value is a long if value.equals(Long.valueOf(value).toString())
* A JSON value is a decimal if it's a JSON number that matches neither
of the above two rules
That should cover most typical number values in a very natural way.
Cases like a 123 value that's explicitly typed as a decimal instead of
a long should be handled with explicit typing information as discussed
below.
I'd be OK with us explicitly *not* supporting special cases like
infinities and NaN values. We'd just throw ValueFormatExceptions for
them on the oak-jcr level and IllegalArgumentExceptions on the
oak-core (or do something similar). Alternatively we should use
explicit typing information with a well defined syntax for expressing
such special cases.
> Finally for binary, date, name, path, reference, weakreference and uri
> there is no direct correspondence in JSON.
Right. There's an additional constraint for binary values in that the
MicroKernel garbage collector needs some way to connect JSON
properties to referenced binaries. It would be useful if the same
convention was used also higher up the stack.
> The way I solved this in spi2microkernel [1] is by encoding values by
> serializing them to their string representation (Value.getString()) and
> prepend its property type (value.getType) in radix 16 and a colon (:).
Sounds like a workable solution, though I have some reservations:
* The explicit encoding of numeric constants from JCR seems a bit
troublesome and makes potential extensions more cumbersome.
* The overloading of normal strings requires that all string values
will need to be checked for whether they need to be escaped.
An alternative solution would be to use something like the @TypeHint
feature used by the JSON functionality in Sling. Instead of "@", we
should use something like "::" that's invalid in a JCR name to prevent
conflicts. With such a solution the example JSON object would look
like this:
"example":{
"long":123,
"another long":"124",
"another long::TypeHint":"long",
"double":"123.4",
"double::TypeHint":"double",
"string":"foo",
"another string":"a:string",
"another string::TypeHint":"string"
}
That's a bit verbose, so we could also put the type hint directly into
the relevant property name, like this:
"example":{
"long":123,
"another long::long":"124",
"double::double":"123.4",
"string":"foo",
"another string::string":"a:string"
}
The main downsides of this approach are:
* Name-based property accesses will potentially need to traverse
through all properties to find a matching name. That should be
manageable since the implementation can pre-scan all property names
and split them to name and type parts.
* There's a potential for conflicts like when a JSON object contains
both "x" and "x::long" properties. That can be dealt with in a commit
validator that prevents such objects from being persisted.
> On a related note: what kind of values do we want to expose from oak-core?
> JSON like or JCR like?
I'd ideally like to keep it JSON-like so we can easily implement a
JavaScript-friendly HTTP mapping directly based on the Oak API without
having to go through extra levels of mapping.
> Implementation wise, would that en/decoding happen inside oak-jcr or oak-core?
I'd put the JSON-JCR type mapping into a shared helper class in
oak-core since it'll be needed by a lot of things like query and node
type handling inside oak-core. But the API interfaces should IMO be
based on JSON types to support cases where JCR typing isn't needed or
wanted.
BR,
Jukka Zitting
Re: Encoding of JCR values in json
Posted by Angela Schreiber <an...@adobe.com>.
hi michael
> Ok, makes sense. We need then
>
> - Have a respective value entity on the oak-api. There is already the
> CoreValue class which should somehow fit/be fitted into the picture.
>
> - Update all places in the API where Scalar is used currently with the
> new value representation.
rather do it now that later, when its used all over the place.
> - Still come up with a encoding into JSON. However since this is not
> exposed through the oak-api we are more flexible here.
yes, we will in any case need to have an encoding into JSON.
> Could you come up with a draft for such a class/interface?
i can give it a try next week. today i wanted to look at
the editor stuff again.
> That would allow me to take the role of the API critic for once ;-)
lol :) we should do that more often anyway, should't we?
regards
angela
> Michael
>
>>
>> kind regards
>> angela
>>
>>> Implementation wise, would that
>>> en/decoding happen inside oak-jcr or oak-core?
>>>
>>> Michael
>>>
>>>
>>> [1]
>>> http://svn.apache.org/viewvc/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/Values.java?view=markup
>>>
Re: Encoding of JCR values in json
Posted by Michael Dürig <md...@apache.org>.
On 12.4.12 18:43, Angela Schreiber wrote:
> hi michael
>
>> On a related note: what kind of values do we want to expose from
>> oak-core? JSON like or JCR like?
>
> i would prefer JCR like on the oak-API level and only have an
> intermediate JSON serialization if there was remoting involved
> in the communication between oak-jcr and oak-core (don't think
> this is the standard setup).
>
> the main reason for that is that we want to push the validation
> and query into the oak-core layer. so imo the oak-core layer
> needs to be aware of property types and therefore i don't
> see an urgent need for extra conversion of values on that layer.
> thinking for example of property definitions (nt),
> value constraints, restrictions within access control entries
> or specific queries.
Ok, makes sense. We need then
- Have a respective value entity on the oak-api. There is already the
CoreValue class which should somehow fit/be fitted into the picture.
- Update all places in the API where Scalar is used currently with the
new value representation.
- Still come up with a encoding into JSON. However since this is not
exposed through the oak-api we are more flexible here.
Could you come up with a draft for such a class/interface? That would
allow me to take the role of the API critic for once ;-)
Michael
>
> kind regards
> angela
>
>> Implementation wise, would that
>> en/decoding happen inside oak-jcr or oak-core?
>>
>> Michael
>>
>>
>> [1]
>> http://svn.apache.org/viewvc/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/Values.java?view=markup
>>
Re: Encoding of JCR values in json
Posted by Angela Schreiber <an...@adobe.com>.
hi michael
> On a related note: what kind of values do we want to expose from
> oak-core? JSON like or JCR like?
i would prefer JCR like on the oak-API level and only have an
intermediate JSON serialization if there was remoting involved
in the communication between oak-jcr and oak-core (don't think
this is the standard setup).
the main reason for that is that we want to push the validation
and query into the oak-core layer. so imo the oak-core layer
needs to be aware of property types and therefore i don't
see an urgent need for extra conversion of values on that layer.
thinking for example of property definitions (nt),
value constraints, restrictions within access control entries
or specific queries.
kind regards
angela
> Implementation wise, would that
> en/decoding happen inside oak-jcr or oak-core?
>
> Michael
>
>
> [1]
> http://svn.apache.org/viewvc/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/Values.java?view=markup