You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by GitBox <gi...@apache.org> on 2022/12/06 04:15:25 UTC
[GitHub] [incubator-kvrocks] simplechen created a discussion: kvrocks 是否有支持json计划
GitHub user simplechen created a discussion: kvrocks 是否有支持json计划
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues.
### Motivation
打扰一下，kvrocks是否有类似rejson的功能 支持计划，打扰了
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] git-hulk added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Any plan to support the JSON in kvrocks?
ReJSON is expected to support in 2023 roadmap: https://github.com/orgs/apache/projects/199?pane=issue&itemId=19474487
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-5612435
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] mapleFU added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
GitHub user mapleFU added a comment to the discussion: Any plan to support the JSON in kvrocks?
## RedisJSON
### Type System
Complex Types:
- Array
- Object
Basic Types:
- Int: a 64-bit signed integer number
- Double: float64 number
- Bool
- String
Bottom Type:
- Null
### Format
#### Storage
Just stores "Raw IJSON" values for json. It's a cbor-like typed json binary format.
When stored in rdb, the "Raw IJSON" format would be cast to "JSON Text".
#### Memory
- serde_json: **User input** value will be wrapped in SerdeJson
- ijson: ijson is an in-place replacement for serde_json, it provides light-weight in-memory format. The storage value will be deserialized with ijson format.
### Commands
Commands are Listed below:
- https://github.com/RedisJSON/RedisJSON/blob/master/redis_json/src/lib.rs#L190-L214
#### Path
- JsonPath
- Require Path and Filter Expression
- Legacy Path Syntax
- Use `a.b` `a["b"]`
# Kvrocks Design
## Method 1: Integrate or Reuse RedisJSON
RedisJson can not be well-optimized if "string" representation is used. Instead, we should pay some efforts to re-implement the old Legact Path Syntax.
If we just reuse RedisJSON, we can:
1. Make RedisJSON supports redis modules
2. Maintain a fork of RedisJSON, and replace redismodules interface with kvrocks' callback
### Pros:
- Can always be compatible with RedisJson
- Easy to integrate Redisearch
- Not need to maintain "Legacy Path Syntax" in kvrocks' code
### Cons:
- It's hard to encapsulate kvrocks's content into Redis context.
- If fork RedisJson, we need to maintain a fork for RedisJSON, and currently we just need to maintain get/set /event/redis-protocol-version and rdb. But if it requires more features, maintaining a fork might be hard.
- The underlying storage format might change, and RedisJSON doesn't have KeyMeta in storage, if RedisJSON or ijson change the underlying implemention
## Method 2: Re-implement JSON in kvrocks
Re-implement JSON in kvrocks is not hard, we need to consider the case below:
- Json Path implementions
- Storage format
- In-memory format
- Thirdparty libraries
## Thirdparty libraries
- Jsoncons: https://github.com/danielaparker/jsoncons
- Support JSON Path, and has cbor, bson extensions
- Not support querying objects directly on binary / text
- SIMDJson:
- Query json directly on raw text
## Storage Format
### Raw Text Data
Pros:
1. Easy to use
2. Can be well compacted
3. Human readable
Cons:
- May consume lots of memory
### Cbor
Pros:
1. Easy to use
2. Usally smaller than json text
3. Strong typed
Cons:
- May complex during parsing
### Jsonb
Pros:
1. It can be fast when reading, because it supports fast seek
2. Strong typed
Cons:
- May complex during parsing
- Might be larger than json text
## In-memory format
- Memory format can be the same as storage format or `jsoncons::json`. The latter might be memory-consuming
- When updating or querying:
- When updating, we might need to support casting it to `jsoncons::json`, and update it, then casting it back to storage format
- When querying, we will use storage format if it's possible
### Corner cases
#### Huge JSON Value
If the json value is small, using all kinds of data is OK, if json value is huge, we need to consider that:
- Would it be heavy when parsing or updating?
- Should it be separate in kvrocks/rocksdb?
### Pros:
- Might have better performance if we can use jsonb without parsing the object
- If the json is huge, we're able to separate it into different parts
### Cons:
- Need lots of compatibility works
- Might not have good performance the first time, and the underlying storage format might change
## References
1. PostgreSQL JSON / JSONB
2. RedisJson
3. Redis Modules
4. Jsoncons
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-6205300
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] simplechen edited a discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user simplechen edited a discussion: Any plan to support the JSON in kvrocks?
As the title
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] simplechen added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user simplechen added a comment to the discussion: Any plan to support the JSON in kvrocks?
> Hi @simplechen. Do you mean to support operations like in MongoDB?
no,mogodb is so complex,like redis json https://redis.io/docs/stack/json/
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-4321716
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] git-hulk added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Any plan to support the JSON in kvrocks?
Yes, feedback is welcome if you have any thoughts.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-4322604
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] simplechen added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user simplechen added a comment to the discussion: Any plan to support the JSON in kvrocks?
Thank you, I see that there is a standardized JSON in the 3.0 plan, and I am looking forward to it
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-4321789
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] mapleFU added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
GitHub user mapleFU added a comment to the discussion: Any plan to support the JSON in kvrocks?
Okay, I'll try to draft one without legacy path, using `jsoncons::json` in memory. The backend implemention can switch from cbor and json text.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-6207523
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] git-hulk added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Any plan to support the JSON in kvrocks?
Should be like Redis JSON: https://redis.io/docs/stack/json/
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-4320883
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] simplechen edited a discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user simplechen edited a discussion: Any plan to support the JSON in kvrocks?
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] torwig added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user torwig added a comment to the discussion: Any plan to support the JSON in kvrocks?
Hi @simplechen.
Do you mean to support operations like in MongoDB?
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-4320577
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] mapleFU added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
GitHub user mapleFU added a comment to the discussion: Any plan to support the JSON in kvrocks?
I can also implement the logic same as IValue and jsonpath in kvrocks, but I guess it's a bit time-consuming.
It could be fast to just using `jsoncons` in memory, and parse it when every time get/set is called, but I guess the performance would be pool
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-6205748
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] simplechen edited a discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user simplechen edited a discussion: Any plan to support the JSON in kvrocks?
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues.
### Motivation
打扰一下，kvrocks是否有类似rejson的功能 支持计划，打扰了
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] mapleFU added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
GitHub user mapleFU added a comment to the discussion: Any plan to support the JSON in kvrocks?
@PragmaTwice @git-hulk
I've listed some possible ways to implement RedisJSON in kvrocks, would you mind take a look?
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-6205302
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] xiaobiaozhao added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user xiaobiaozhao added a comment to the discussion: Any plan to support the JSON in kvrocks?
I think if you need to store json strings, consider using lua scripts instead, as cjson codec support is available in lua vm
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-4323241
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] git-hulk added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
GitHub user git-hulk added a comment to the discussion: Any plan to support the JSON in kvrocks?
@mapleFU Thanks for your great proposals.
For the implementation method, I prefer to re-implement in Kvrocks which is more coherent than use the redis module. As the formatting, jsonb is a good choice if it wouldn’t bring too much work while comparing to the raw text. And for the performance, I don’t worry much about this since we can try to improve if users did suffer from the performance issues.
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-6206002
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] simplechen added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user simplechen added a comment to the discussion: Any plan to support the JSON in kvrocks?
thanks
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-5612385
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] simplechen added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user simplechen added a comment to the discussion: Any plan to support the JSON in kvrocks?
yes,like rejson,
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-4321657
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
[GitHub] [incubator-kvrocks] simplechen added a comment to the discussion: Any plan to support the JSON in kvrocks?
Posted by GitBox <gi...@apache.org>.
GitHub user simplechen added a comment to the discussion: Any plan to support the JSON in kvrocks?
Is json data type support in our plan, thank you
GitHub link: https://github.com/apache/incubator-kvrocks/discussions/1164#discussioncomment-4321693
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org