You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2019/12/26 02:06:39 UTC
[GitHub] [skywalking] wu-sheng opened a new issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
wu-sheng opened a new issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131
@tristaZero
I created this to track the thing you are working one.
To All
We will add ShardingSphere proxy configuration file with documentation to guide users to use ShardingSphere + MySQL to support larger scale system monitoring.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587302393
Hi everyone, after discussion on dev@list, here is the schedule of this issue,
- [x] Acquire all required tables and query SQLs from db logs, like general log of MySQL.
- [ ] Analyse those tables and query SQLs to explore appropriate sharding strategy.
- [ ] Provide class implementation of `PreciseShardingAlgorithm` and `RangeShardingAlgorithm` of ShardingSphere to do sharding.
- [ ] Provide configuration of Sharding Proxy[2] for Skywalking.
- [ ] Provide e2e tests of Skywalking.
@dmsolr will be in charge of 4th and 5th, and i will take care of 1st to 3rd of this list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-583974818
I think the configuration should be placed in `install/shardingsphere/proxy/my.cnf`. At the same time, the following changes are expected
1. [readme](https://github.com/apache/skywalking/tree/master/install) should be updated
1. [storage doc](https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md) should be updated.
1. e2e test should be changed. A tip, based on this PR https://github.com/apache/skywalking/pull/4335/files#diff-5c4acdc03cc5bf967f23c6c3564a8cb7, the mysql download path is not right. We are trying to fix in PR #4335, you should notice at yours too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584533083
@wu-sheng Here is another news,
ShardingSphere is planning to do some optimization for sharding algorithm. And the main idea is to provide user some `default sharding algorithms`, therefore there is no need of `sharding-algorithm.jar`, and user just chooses one of `existing algorithms` and configure it in `my.cnf`.
I am not sure the concrete date for its completion, but once it finishes, i will continue to update this issue here.
I would like to hear you suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584533083
@wu-sheng Here is another news,
ShardingSphere is planning to do some optimization for sharding algorithm. And the main idea is to provide user some `default sharding algorithms`, therefore there is no need of `sharding-algorithm.jar`, and user just chooses one of `existing algorithms` and configure it in `my.cnf`.
I am not sure the concrete date for its completion, but don't worry, once it finishes, i will continue to update this issue here.
I would like to hear you suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588127418
> I don't like the idea we config them one by one. We could be use which tables are not sharded. Segment is a special case, and all others are metrics or related, so we could use the same rule for them.
You mean config 282 tables at once?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-593278342
@wu-sheng Hi, I want to double-check two issues, could you give me some hints?
1. The configuration `my.cnf` of ShardingProxy is supposed to locate in `root_dir/install/shardingsphere/proxy/` to provide user or e2e test, right?
2. The directory of `root_dir/test/e2e/e2e-shardingsphere` fits for ShardingSphere e2e ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-590027379
> Based on the description, if there is a new dynamic table we missed on the config file, the system still works, this is the risk.
To some extend, you are right. However it is human being's mistake that table config is missing, rather than program which just needs to look at the configuraion whether it is correct or not. Even we missed some sharding table rule, SS can share one DB's stress as well. Imagining, if there is no sharding rule, all the tables will be put into one DB.
On the other hand, indeed configuration of `user_*` can reduce the likelihood of mistakes and SS should shadow more configuration detail and simply user's work and understanding.
Currently, i am sorry that SS can not do that well, but what you worried also gave me some new ideas to make it better. Thanks very much.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588049040
> What you concerned is to provide certain rules for tables without sharding, right?
No, the concern is table could be added without source code changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587314746
### All the tables created by SW
alarm_record
all_heatmap
all_heatmap_day
all_heatmap_hour
all_heatmap_month
all_p50
all_p50_day
all_p50_hour
all_p50_month
all_p75
all_p75_day
all_p75_hour
all_p75_month
all_p90
all_p90_day
all_p90_hour
all_p90_month
all_p95
all_p95_day
all_p95_hour
all_p95_month
all_p99
all_p99_day
all_p99_hour
all_p99_month
database_access_cpm
database_access_cpm_day
database_access_cpm_hour
database_access_cpm_month
database_access_p50
database_access_p50_day
database_access_p50_hour
database_access_p50_month
database_access_p75
database_access_p75_day
database_access_p75_hour
database_access_p75_month
database_access_p90
database_access_p90_day
database_access_p90_hour
database_access_p90_month
database_access_p95
database_access_p95_day
database_access_p95_hour
database_access_p95_month
database_access_p99
database_access_p99_day
database_access_p99_hour
database_access_p99_month
database_access_resp_time
database_access_resp_time_day
database_access_resp_time_hour
database_access_resp_time_month
database_access_sla
database_access_sla_day
database_access_sla_hour
database_access_sla_month
endpoint_avg
endpoint_avg_day
endpoint_avg_hour
endpoint_avg_month
endpoint_cpm
endpoint_cpm_day
endpoint_cpm_hour
endpoint_cpm_month
endpoint_inventory
endpoint_p50
endpoint_p50_day
endpoint_p50_hour
endpoint_p50_month
endpoint_p75
endpoint_p75_day
endpoint_p75_hour
endpoint_p75_month
endpoint_p90
endpoint_p90_day
endpoint_p90_hour
endpoint_p90_month
endpoint_p95
endpoint_p95_day
endpoint_p95_hour
endpoint_p95_month
endpoint_p99
endpoint_p99_day
endpoint_p99_hour
endpoint_p99_month
endpoint_relation_cpm
endpoint_relation_cpm_day
endpoint_relation_cpm_hour
endpoint_relation_cpm_month
endpoint_relation_resp_time
endpoint_relation_resp_time_day
endpoint_relation_resp_time_hour
endpoint_relation_resp_time_month
endpoint_relation_server_side
endpoint_relation_server_side_day
endpoint_relation_server_side_hour
endpoint_relation_server_side_month
endpoint_sla
endpoint_sla_day
endpoint_sla_hour
endpoint_sla_month
envoy_heap_memory_max_used
envoy_heap_memory_max_used_day
envoy_heap_memory_max_used_hour
envoy_heap_memory_max_used_month
envoy_parent_connections_used
envoy_parent_connections_used_day
envoy_parent_connections_used_hour
envoy_parent_connections_used_month
envoy_total_connections_used
envoy_total_connections_used_day
envoy_total_connections_used_hour
envoy_total_connections_used_month
http_access_log
instance_clr_available_completion_port_threads
instance_clr_available_completion_port_threads_day
instance_clr_available_completion_port_threads_hour
instance_clr_available_completion_port_threads_month
instance_clr_available_worker_threads
instance_clr_available_worker_threads_day
instance_clr_available_worker_threads_hour
instance_clr_available_worker_threads_month
instance_clr_cpu
instance_clr_cpu_day
instance_clr_cpu_hour
instance_clr_cpu_month
instance_clr_gen0_collect_count
instance_clr_gen0_collect_count_day
instance_clr_gen0_collect_count_hour
instance_clr_gen0_collect_count_month
instance_clr_gen1_collect_count
instance_clr_gen1_collect_count_day
instance_clr_gen1_collect_count_hour
instance_clr_gen1_collect_count_month
instance_clr_gen2_collect_count
instance_clr_gen2_collect_count_day
instance_clr_gen2_collect_count_hour
instance_clr_gen2_collect_count_month
instance_clr_heap_memory
instance_clr_heap_memory_day
instance_clr_heap_memory_hour
instance_clr_heap_memory_month
instance_clr_max_completion_port_threads
instance_clr_max_completion_port_threads_day
instance_clr_max_completion_port_threads_hour
instance_clr_max_completion_port_threads_month
instance_clr_max_worker_threads
instance_clr_max_worker_threads_day
instance_clr_max_worker_threads_hour
instance_clr_max_worker_threads_month
instance_jvm_cpu
instance_jvm_cpu_day
instance_jvm_cpu_hour
instance_jvm_cpu_month
instance_jvm_memory_heap
instance_jvm_memory_heap_day
instance_jvm_memory_heap_hour
instance_jvm_memory_heap_max
instance_jvm_memory_heap_max_day
instance_jvm_memory_heap_max_hour
instance_jvm_memory_heap_max_month
instance_jvm_memory_heap_month
instance_jvm_memory_noheap
instance_jvm_memory_noheap_day
instance_jvm_memory_noheap_hour
instance_jvm_memory_noheap_max
instance_jvm_memory_noheap_max_day
instance_jvm_memory_noheap_max_hour
instance_jvm_memory_noheap_max_month
instance_jvm_memory_noheap_month
instance_jvm_old_gc_count
instance_jvm_old_gc_count_day
instance_jvm_old_gc_count_hour
instance_jvm_old_gc_count_month
instance_jvm_old_gc_time
instance_jvm_old_gc_time_day
instance_jvm_old_gc_time_hour
instance_jvm_old_gc_time_month
instance_jvm_young_gc_count
instance_jvm_young_gc_count_day
instance_jvm_young_gc_count_hour
instance_jvm_young_gc_count_month
instance_jvm_young_gc_time
instance_jvm_young_gc_time_day
instance_jvm_young_gc_time_hour
instance_jvm_young_gc_time_month
jaeger_span
network_address_inventory
register_lock
segment
service_cpm
service_cpm_day
service_cpm_hour
service_cpm_month
service_instance_cpm
service_instance_cpm_day
service_instance_cpm_hour
service_instance_cpm_month
service_instance_inventory
service_instance_resp_time
service_instance_resp_time_day
service_instance_resp_time_hour
service_instance_resp_time_month
service_instance_sla
service_instance_sla_day
service_instance_sla_hour
service_instance_sla_month
service_inventory
service_p50
service_p50_day
service_p50_hour
service_p50_month
service_p75
service_p75_day
service_p75_hour
service_p75_month
service_p90
service_p90_day
service_p90_hour
service_p90_month
service_p95
service_p95_day
service_p95_hour
service_p95_month
service_p99
service_p99_day
service_p99_hour
service_p99_month
service_relation_client_call_sla
service_relation_client_call_sla_day
service_relation_client_call_sla_hour
service_relation_client_call_sla_month
service_relation_client_cpm
service_relation_client_cpm_day
service_relation_client_cpm_hour
service_relation_client_cpm_month
service_relation_client_p50
service_relation_client_p50_day
service_relation_client_p50_hour
service_relation_client_p50_month
service_relation_client_p75
service_relation_client_p75_day
service_relation_client_p75_hour
service_relation_client_p75_month
service_relation_client_p90
service_relation_client_p90_day
service_relation_client_p90_hour
service_relation_client_p90_month
service_relation_client_p95
service_relation_client_p95_day
service_relation_client_p95_hour
service_relation_client_p95_month
service_relation_client_p99
service_relation_client_p99_day
service_relation_client_p99_hour
service_relation_client_p99_month
service_relation_client_resp_time
service_relation_client_resp_time_day
service_relation_client_resp_time_hour
service_relation_client_resp_time_month
service_relation_client_side
service_relation_client_side_day
service_relation_client_side_hour
service_relation_client_side_month
service_relation_server_call_sla
service_relation_server_call_sla_day
service_relation_server_call_sla_hour
service_relation_server_call_sla_month
service_relation_server_cpm
service_relation_server_cpm_day
service_relation_server_cpm_hour
service_relation_server_cpm_month
service_relation_server_p50
service_relation_server_p50_day
service_relation_server_p50_hour
service_relation_server_p50_month
service_relation_server_p75
service_relation_server_p75_day
service_relation_server_p75_hour
service_relation_server_p75_month
service_relation_server_p90
service_relation_server_p90_day
service_relation_server_p90_hour
service_relation_server_p90_month
service_relation_server_p95
service_relation_server_p95_day
service_relation_server_p95_hour
service_relation_server_p95_month
service_relation_server_p99
service_relation_server_p99_day
service_relation_server_p99_hour
service_relation_server_p99_month
service_relation_server_resp_time
service_relation_server_resp_time_day
service_relation_server_resp_time_hour
service_relation_server_resp_time_month
service_relation_server_side
service_relation_server_side_day
service_relation_server_side_hour
service_relation_server_side_month
service_resp_time
service_resp_time_day
service_resp_time_hour
service_resp_time_month
service_sla
service_sla_day
service_sla_hour
service_sla_month
top_n_database_statement
zipkin_span
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588582694
> I mean config the table names with some rules(Regex), or exclusion rule is better. Such as `all table exclude tableA, tableB, tableC` are using this sharding rule.
Those tables need to be configured in [sharding-proxy-config](https://shardingsphere.apache.org/document/current/en/manual/sharding-proxy/configuration/), which means we have to configure tables one by one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588034509
My first idea of sharding stratergy is to shard certain table, taking `alarm_record` for example, into 4 DBs and 10 tables each DB, i.e, totally 40 sub-tables for one table with the sharding column `time_bucket`. Other tables at the frist can adopt the same sharding stratergy with their own sharding columns.
What do you think? @dmsolr @wu-sheng
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-593429451
> The configuration my.cnf of ShardingProxy is supposed to locate in root_dir/install/shardingsphere/proxy/ to provide user or e2e test, right?
This is for end users and documentation. Of course, if this is as same as the e2e requirement, you could put it there too.
> The directory of root_dir/test/e2e/e2e-shardingsphere fits for ShardingSphere e2e ?
Yes, as a maven module.
> Hi, I have no clear way how to add an e2e test for e2e-ShardingSphere after reading #4335 and e2e-mysql.
#4335 is a totally new case, so, I prefer you following the existing e2e-mysql case.
> My initial idea is to copy the dir of e2e-mysql to e2e-ShardingSphere and modify rc0-prepare.sh to run Sharding Proxy as backend server. Do you think it is feasible or not?
Make sense to me. The only addition is, you may need to add multiple MySQL server startup script too. Considering the database instance number, please recheck whether you could share the `my.cnf`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588134088
I mean config the table names with some rules(Regex), or exclusion rule is better. Such as `all table exclude tableA, tableB, tableC` are using this sharding rule.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-590066045
@tristaZero Please feel free to go by following your existing plan. All I asked are about I want the ss-proxy solution more practical than just a documentation. Right now, some key features are missing, people are hard to follow no matter what kind of documents we wrote.
Is that possible to provide these two features(restrict rule for no rule table, wildcard name) quickly? And included in the next release?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588029909
> Most of SkyWalking tables are created by OAL scripts. Do you know what is it?
>
> Limited tables are the inventory tables, or we called metadata tables, or not time serious tables.
What you concerned is to provide certain rules for tables without sharding, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588105831
> My first idea of sharding stratergy is to shard certain table, taking alarm_record for example, into 4 DBs and 10 tables each DB
The number of shards is not the issue. My point is targeting which tables should be sharded. I don't like the idea we config them one by one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588105831
> My first idea of sharding stratergy is to shard certain table, taking alarm_record for example, into 4 DBs and 10 tables each DB
The number of shards is not the issue. My point is targeting which tables should be sharded. I don't like the idea we config them one by one. We could be use which tables are not sharded. Segment is a special case, and all others are metrics or related, so we could use the same rule for them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] dmsolr commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
dmsolr commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588135131
> Could you give me some hints how to handle this issue?
`ModelInstaller` will create a table for each one. Includes generated OAL script.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584523278
Hi @wu-sheng
Not only `my.cnf`, a `sharding-algorithm.jar` is necessary to be put into `install/shardingsphere/proxy/` to handle sharding tables and dbs.
If user plan to adopt sharding proxy as backend server, he has to download sharding proxy and copy `my.cnf`, and `sharding-algorithm.jar` from `install/shardingsphere/proxy`to specified dir.
Do you think the above solution is feasible? And then how to specify my workload? Here is a task list from my understanding,
1. Provide `my.cnf`, a `sharding-algorithm.jar` and put them into `install/shardingsphere/proxy/`
2.Update `readme`
3.Update `storage doc`
4.Create e2e test referring to `test/e2e/mysql`
If i came to wrong conclusion, please be clear what i should do next. Thanks in advance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588582694
> I mean config the table names with some rules(Regex), or exclusion rule is better. Such as `all table exclude tableA, tableB, tableC` are using this sharding rule.
Those tables need to be configured in [sharding-proxy-config](https://shardingsphere.apache.org/document/current/en/manual/sharding-proxy/configuration/), which means we have to configure tables one by one.
The configuration example is below,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584525495
> `sharding-algorithm.jar` needs to be taken care of differently. It is a binary distribution, so it can't be hosted in any repo.
@wu-sheng Agree with you, and it is also what i am confused and have no idea to handle. I can provide a `sharding-algorithm.java`, but use need to complie it to generate `sharding-algorithm.jar`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-589500912
> Can't you use a Regex or expression to represent the tables? I was assuming we could do that.
Using a Regex or expression to represent the tables on `configuration of sharding proxy`?
@wu-sheng
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588582921
```
schemaName: sharding_db
dataSources:
ds0:
url: jdbc:postgresql://localhost:5432/ds0
username: root
password:
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 65
ds1:
url: jdbc:postgresql://localhost:5432/ds1
username: root
password:
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 65
shardingRule:
**tables**:
**t_order**:
actualDataNodes: ds${0..1}.t_order${0..1}
databaseStrategy:
inline:
shardingColumn: user_id
algorithmExpression: ds${user_id % 2}
tableStrategy:
inline:
shardingColumn: order_id
algorithmExpression: t_order${order_id % 2}
keyGenerator:
type: SNOWFLAKE
column: order_id
**t_order_item**:
actualDataNodes: ds${0..1}.t_order_item${0..1}
databaseStrategy:
inline:
shardingColumn: user_id
algorithmExpression: ds${user_id % 2}
tableStrategy:
inline:
shardingColumn: order_id
algorithmExpression: t_order_item${order_id % 2}
keyGenerator:
type: SNOWFLAKE
column: order_item_id
bindingTables:
- t_order,t_order_item
defaultTableStrategy:
none:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588582921
schemaName: sharding_db
dataSources:
ds0:
url: jdbc:postgresql://localhost:5432/ds0
username: root
password:
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 65
ds1:
url: jdbc:postgresql://localhost:5432/ds1
username: root
password:
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 65
shardingRule:
tables:
**t_order**:
actualDataNodes: ds${0..1}.t_order${0..1}
databaseStrategy:
inline:
shardingColumn: user_id
algorithmExpression: ds${user_id % 2}
tableStrategy:
inline:
shardingColumn: order_id
algorithmExpression: t_order${order_id % 2}
keyGenerator:
type: SNOWFLAKE
column: order_id
**t_order_item**:
actualDataNodes: ds${0..1}.t_order_item${0..1}
databaseStrategy:
inline:
shardingColumn: user_id
algorithmExpression: ds${user_id % 2}
tableStrategy:
inline:
shardingColumn: order_id
algorithmExpression: t_order_item${order_id % 2}
keyGenerator:
type: SNOWFLAKE
column: order_item_id
bindingTables:
- t_order,t_order_item
defaultTableStrategy:
none:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-590192000
> How to make proxy know there is a to-check table which is not existing in configuration?
You don't need that. You just need to provide a feature about, whether table without config rule, should go into the default datasource or report error. Or, support no default datasource.
> I am not sure whether it will be included in next release, IMO it is not a widespread need that all of the user_* tables, i.e, user_all, user_single need sharding. Most cases i got are to shard tables like t_order and t_order_item, at this moment wildcard is not needed.
I am not following, why setting `t_order*` for both `t_order` and `t_order_item` is not reasonable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587390211
> Another reminder is, Could you use a consistent-hash-func(table_name) and an exclusion rule for limited tables
Thanks for your reminder, however i could not get your point. How to understand to use `consistent-hash-func(table_name) ` to provide the sharding configuration once for all? What is the meaning of `an exclusion rule for limited tables`. @wu-sheng
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-589526437
OK. Then the question will be, how to check the configuration is still good for every commit in the e2e? How to check we are not missing any table? Could ShardingProxy start up in the strict way? For strict way, it means, if a SQL can't find the sharding rule, it will fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588582694
> I mean config the table names with some rules(Regex), or exclusion rule is better. Such as `all table exclude tableA, tableB, tableC` are using this sharding rule.
Those tables need to be configured in [sharding-proxy-config](https://shardingsphere.apache.org/document/current/en/manual/sharding-proxy/configuration/), which means we have to configure tables one by one. However we can generate this configuration by means of `some script written by us`, which depends on @dmsolr 's preference.
The configuration example is below,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584534107
> There are some options
>
> 1. As you said, you could provide the source codes hosted in SkyWalking main repo, then, it will be released as a separate jar in 7.0.0. You could update the document to point to the jar, even it is not available today.
>
> 2. Open a new repo to host it and ask for a release process for it. As you haven't been a committer/PMC, you need one to help you with this.
>
>
> Which one do you want?
First one sounds easy to make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587390211
> Another reminder is, Could you use a consistent-hash-func(table_name) and an exclusion rule for limited tables
Thanks for your reminder, however i could not get your point. How to use `consistent-hash-func(table_name) ` to provide the sharding configuration once for all? What is the meaning of `an exclusion rule for limited tables`. @wu-sheng
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-589933024
SS find sharding rule of one table completely based on config, i.e, if we miss any table, SS will consider it as `non-sharding table` existed on `default-datasource`(Need to configure), which SS sends the query to. When starting up, SS will load all sub-tables configured at configuration and check their metadata one by one.
@wu-sheng
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-589949730
Based on the description, if there is a new dynamic table we missed on the config file, the system still works, this is the risk. Because, eventually, the default datasource database crashes without warning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584525495
> `sharding-algorithm.jar` needs to be taken care of differently. It is a binary distribution, so it can't be hosted in any repo.
@wu-sheng Agree with you, and it is also what i am confused and have no idea to deal with it. I can provide a `sharding-algorithm.java`, but user needs to complie it to generate `sharding-algorithm.jar`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-589140762
Can't you use a Regex or expression to represent the tables? I was assuming we could do that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587315261
### Some Query SQLs
select count(1) total alarm_record where 1=1 and time_bucket >= 20191210165700 and time_bucket <= 20191210171299 order by start_time desc
select * from alarm_record where 1=1 and time_bucket >= ? and time_bucket <= ? order by start_time desc LIMIT 0, 20
select avg(value) value,entity_id from endpoint_avg where time_bucket >= ? and time_bucket <= ? group by entity_id order by value desc limit 10
select count(1) total from segment where 1=1 and time_bucket >= ? and time_bucket <= ? order by latency desc
select * from segment where 1=1 and time_bucket >= 20191210165700 and time_bucket <= 20191210171299 order by latency desc LIMIT 0, 15
select entity_id id, avg(value) value from service_cpm where (entity_id in ('2','3')) and time_bucket>= ? and time_bucket<=? group by entity_id
select avg(value) value,entity_id from service_cpm where time_bucket >= ? and time_bucket <= ? group by entity_id order by value desc limit 10
select id, value from endpoint_p90 where id in ('201912101657','201912101658','201912101659','201912101700','201912101701','201912101702','201912101703','201912101704','201912101705','201912101706','201912101707','201912101708','201912101709','201912101710','201912101711','201912101712')
select id, value from instance_clr_cpu where id in ('201912101657_2','201912101658_2','201912101659_2','201912101700_2','201912101701_2','201912101702_2','201912101703_2','201912101704_2','201912101705_2','201912101706_2','201912101707_2','201912101708_2','201912101709_2','201912101710_2','201912101711_2','201912101712_2')
select id, value from service_p50 where id in ('201912101657_2','201912101658_2','201912101659_2','201912101700_2','201912101701_2','201912101702_2','201912101703_2','201912101704_2','201912101705_2','201912101706_2','201912101707_2','201912101708_2','201912101709_2','201912101710_2','201912101711_2','201912101712_2')
select entity_id from service_relation_client_side where time_bucket>= 201912101657 and time_bucket<=201912101712 and (source_service_id=2 or dest_service_id=2 ) group by entity_id
select entity_id id, avg(value) value from service_resp_time where (entity_id in ('2','3')) and time_bucket>= 201912101657 and time_bucket<=201912101712 group by entity_id
select entity_id id, avg(percentage) value from service_sla where (entity_id in ('2','3')) and time_bucket>= ? and time_bucket<=? group by entity_id
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588126490
> No, the concern is table could be added without source code changed.
Could you give me some hints how to handle this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584537206
> ShardingSphere is planning to do some optimization for sharding algorithm. And the main idea is to provide user some default sharding algorithms, therefore there is no need of sharding-algorithm.jar, and user just chooses one of existing algorithms and configure it in my.cnf.
It is OK for you about when that comes, you could remove the codes and module from SW.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584524257
If this runs into the release process, it seems you need another repo? If that, I would like to ask you to propose this in the mail list including purpose, scope, repo name etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-590156844
@wu-sheng
> restrict rule
How to make proxy know there is a to-check table which is not existing in configuration?
> wildcard name
I am not sure whether it will be included in next release, IMO it is not a widespread need that all of the `user_*` tables, i.e, user_all, user_single need sharding. Most cases i got are to shard tables like `t_order` and `t_order_item`, at this moment `wildcard` is not needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588582921
```
schemaName: sharding_db
dataSources:
ds0:
url: jdbc:postgresql://localhost:5432/ds0
username: root
password:
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 65
ds1:
url: jdbc:postgresql://localhost:5432/ds1
username: root
password:
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 65
shardingRule:
tables: (Look!)
t_order: (Look!)
actualDataNodes: ds${0..1}.t_order${0..1}
databaseStrategy:
inline:
shardingColumn: user_id
algorithmExpression: ds${user_id % 2}
tableStrategy:
inline:
shardingColumn: order_id
algorithmExpression: t_order${order_id % 2}
keyGenerator:
type: SNOWFLAKE
column: order_id
t_order_item: (Look!)
actualDataNodes: ds${0..1}.t_order_item${0..1}
databaseStrategy:
inline:
shardingColumn: user_id
algorithmExpression: ds${user_id % 2}
tableStrategy:
inline:
shardingColumn: order_id
algorithmExpression: t_order_item${order_id % 2}
keyGenerator:
type: SNOWFLAKE
column: order_item_id
bindingTables:
- t_order,t_order_item
defaultTableStrategy:
none:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-590200076
We keep doing as we planned, all these discussions are improvements. We could set up the table one by one for sure. As I said, all these I asked, are because we want the users will really ss-proxy in the production.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584610355
> Sharding-algorithm implements interface of ShardingSphere, which means ShardingSphere pom will be imported into SkyWalking. Is it ok?
It is imported by `provided` scope only, so, there is nothing to take care of. Don't worry about this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587321026
> From my point, it seems time_bucket and id are feasible for tables to do range sharding.
I can confirm that, `id`(entity_id) is the unique ID, and `time_bucket` is suitable for sharding.
Another reminder is, Could you use a `consistent-hash-func(table_name)` and an exclusion rule for limited tables to provide the sharding configuration once for all, rather than for all tables one by one. The table could be added dynamically.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587314368
For the 2nd one sub-task, i present an analysis for us to discuss, welcome your comment and opinion. Specially ping @wu-sheng and @dmsolr here.
Here are [311 tables](https://github.com/apache/skywalking/issues/4131#issuecomment-587314746) created by sw, and [282 tables](https://github.com/apache/skywalking/issues/4131#issuecomment-587315127) will be targets for sharding (initally supposed) since @wu-sheng told me tables with `all_` starter and `_inventory` terminator and one `register_lock` table are excluded.
After digging the MySQL general log, i picked up [some representative SQLs](https://github.com/apache/skywalking/issues/4131#issuecomment-587315261) for us to discuss. From my point, it seems `time_bucket` and `id` are feasible for tables to do range sharding. I appended all the tables, target tables and SQLs for analysis below for you to review. Looking forward to your suggestion.
In addtion, there are many tables and SQLs to shard, so i suggest at the beginning, we can pick up some important tables and SQLs to do sharding, the aim of which is to check our solution
can run well from A to Z. After that, we can add more tables to `configuration` of `ShardingProxy` to shard continually, which can help split a huge task into many small ones and PRs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587410043
Most of SkyWalking tables are created by OAL scripts. Do you know what is it?
Limited tables are the inventory tables, or we called metadata tables, or not time serious tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] dmsolr commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
dmsolr commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587986612
We have 4 kinds of data(table). They are `record`, `metadata`(inventory + lock), `metrics` and `none stream`. `none stream` is also `record`(currently).
As we know, metadata is not very large in usually. As @wu-sheng said, they don't need to split sharding. The others need to have sharding.
Besides this, most of tables are `metrics`. And they have several kinds of downsampling(month, day, hour, minute and second). By default, the unit is second, so it does not have a time unit as the suffix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-593750362
@wu-sheng Hi Thanks for your useful feedback, and I will try to move forward.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584530629
There are some options
1. As you said, you could provide the source codes hosted in SkyWalking main repo, then, it will be released as a separate jar in 7.0.0. You could update the document to point to the jar, even it is not available today.
1. Open a new repo to host it and ask for a release process for it. As you haven't been a committer/PMC, you need one to help you with this.
Which one do you want?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-593319725
Hi, I have no clear way how to add an e2e test for `e2e-ShardingSphere` after reading #4335 and e2e-mysql. My initial idea is to copy the dir of `e2e-mysql` to `e2e-ShardingSphere` and modify `rc0-prepare.sh` to run `Sharding Proxy` as backend server. Do you think it is feasible or not?
I am looking forward to your kind suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-568949594
@wu-sheng Hi, thanks for this issue which any updates from me would be related to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587390211
> Another reminder is, Could you use a consistent-hash-func(table_name) and an exclusion rule for limited tables
Thanks for your reminder, however i could not get your point. How to use `consistent-hash-func(table_name) ` to provide the sharding configuration once for all? What is the meaning of `an exclusion rule for limited tables`. @wu-sheng
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588582694
> I mean config the table names with some rules(Regex), or exclusion rule is better. Such as `all table exclude tableA, tableB, tableC` are using this sharding rule.
@wu-sheng
Those tables need to be configured in [sharding-proxy-config](https://shardingsphere.apache.org/document/current/en/manual/sharding-proxy/configuration/), which means we have to configure tables one by one. However we can generate this configuration by means of `some script written by us`, which depends on @dmsolr 's preference.
The configuration example is below,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588127418
> I don't like the idea we config them one by one. We could be use which tables are not sharded. Segment is a special case, and all others are metrics or related, so we could use the same rule for them.
You mean configuring 282 tables at once?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587314368
For the 2nd one sub-task, i present an analysis for us to discuss, welcome your comment and opinion. Specially ping @wu-sheng and @dmsolr here.
Here are 311 tables created by sw, and 282 tables will be targets for sharding (initally supposed) since @wu-sheng told me tables with `all_` starter and `_inventory` terminator and one `register_lock` table are excluded.
After digging the MySQL general log, i picked up some representative SQLs for us to discuss. From my point, it seems `time_bucket` and `id` are feasible for tables to do range sharding. I appended all the tables, target tables and SQLs for analysis below for you to review. Looking forward to your suggestion.
In addtion, there are many tables and SQLs to shard, so i suggest at the beginning, we can pick up some important tables and SQLs to do sharding, the aim of which is to check our solution
can run well from A to Z. After that, we can add more tables to `configuration` of `ShardingProxy` to shard continually, which can help split a huge task into many small ones and PRs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-590027379
> Based on the description, if there is a new dynamic table we missed on the config file, the system still works, this is the risk.
To some extend, you are right. However it is human being's mistake that table config is missing, rather than program which just needs to look at the configuraion whether it is correct or not. Even we missed some sharding table rule, SS can share one DB's stress as well. Imagining, if there is no sharding rule, all the tables will be put into one DB.
On the other hand, indeed configuration of `user_*` can reduce the likelihood of mistakes and SS should shadow more configuration detail and simply user's work and understanding.
Currently, i am sorry that SS can not do so well, but what you worried also gave me some new ideas to make it better. Thanks very much.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero edited a comment on issue #4131:
ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero edited a comment on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587315261
### Some Query SQLs
select count(1) total **alarm_record** where 1=1 and **time_bucket** >= 20191210165700 and time_bucket <= 20191210171299 order by start_time desc
select * from **alarm_record** where 1=1 and **time_bucket** >= ? and time_bucket <= ? order by start_time desc LIMIT 0, 20
select avg(value) value,entity_id from **endpoint_avg** where **time_bucket** >= ? and time_bucket <= ? group by entity_id order by value desc limit 10
select count(1) total from **segment** where 1=1 and **time_bucket** >= ? and time_bucket <= ? order by latency desc
select * from **segment** where 1=1 and **time_bucket** >= 20191210165700 and time_bucket <= 20191210171299 order by latency desc LIMIT 0, 15
select entity_id id, avg(value) value from **service_cpm** where (entity_id in ('2','3')) and **time_bucket**>= ? and time_bucket<=? group by entity_id
select avg(value) value,entity_id from **service_cpm** where **time_bucket** >= ? and time_bucket <= ? group by entity_id order by value desc limit 10
select id, value from **endpoint_p90** where **id** in ('201912101657','201912101658','201912101659','201912101700','201912101701','201912101702','201912101703','201912101704','201912101705','201912101706','201912101707','201912101708','201912101709','201912101710','201912101711','201912101712')
select id, value from **instance_clr_cpu** where **id** in ('201912101657_2','201912101658_2','201912101659_2','201912101700_2','201912101701_2','201912101702_2','201912101703_2','201912101704_2','201912101705_2','201912101706_2','201912101707_2','201912101708_2','201912101709_2','201912101710_2','201912101711_2','201912101712_2')
select id, value from **service_p50** where **id** in ('201912101657_2','201912101658_2','201912101659_2','201912101700_2','201912101701_2','201912101702_2','201912101703_2','201912101704_2','201912101705_2','201912101706_2','201912101707_2','201912101708_2','201912101709_2','201912101710_2','201912101711_2','201912101712_2')
select entity_id from **service_relation_client_side** where **time_bucket**>= 201912101657 and time_bucket<=201912101712 and (source_service_id=2 or dest_service_id=2 ) group by entity_id
select entity_id id, avg(value) value from **service_resp_time** where (entity_id in ('2','3')) and time_bucket>= 201912101657 and **time_bucket**<=201912101712 group by entity_id
select entity_id id, avg(percentage) value from **service_sla** where (**entity_id** in ('2','3')) and time_bucket>= ? and **time_bucket**<=? group by entity_id
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-588126490
> No, the concern is table could be added without source code changed.
Could you give me some hints how to handle this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584534922
Then the expected module will be `oap-server/server-storage-plugin/storage-shardingsphere-algorithms`. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-590262740
Got it, therefore i will push it forward following the our existing plan.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584523846
`sharding-algorithm.jar` needs to be taken care of differently. It is a binary distribution, so it can't be hosted in any repo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-584594056
> Then the expected module will be `oap-server/server-storage-plugin/storage-shardingsphere-algorithms`. WDYT?
Sharding-algorithm implements `interface of ShardingSphere`, which means ShardingSphere pom will be imported into SkyWalking. Is it ok? On the other hand, [Second solution](https://github.com/apache/skywalking/issues/4131#issuecomment-584533083) does not encounter this issue.
Wait for the second one? Or try to first one？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-583944522
@wu-sheng Hi, sir. I am considering using ShardingProxy to be the backend storage for sw. If i add a configuration, something like `my.cnf` for MySQL, do you think i should put it into sw project? If so, where is a better place?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-587315127
### Target tables
alarm_record
database_access_cpm
database_access_cpm_day
database_access_cpm_hour
database_access_cpm_month
database_access_p50
database_access_p50_day
database_access_p50_hour
database_access_p50_month
database_access_p75
database_access_p75_day
database_access_p75_hour
database_access_p75_month
database_access_p90
database_access_p90_day
database_access_p90_hour
database_access_p90_month
database_access_p95
database_access_p95_day
database_access_p95_hour
database_access_p95_month
database_access_p99
database_access_p99_day
database_access_p99_hour
database_access_p99_month
database_access_resp_time
database_access_resp_time_day
database_access_resp_time_hour
database_access_resp_time_month
database_access_sla
database_access_sla_day
database_access_sla_hour
database_access_sla_month
endpoint_avg
endpoint_avg_day
endpoint_avg_hour
endpoint_avg_month
endpoint_cpm
endpoint_cpm_day
endpoint_cpm_hour
endpoint_cpm_month
endpoint_p50
endpoint_p50_day
endpoint_p50_hour
endpoint_p50_month
endpoint_p75
endpoint_p75_day
endpoint_p75_hour
endpoint_p75_month
endpoint_p90
endpoint_p90_day
endpoint_p90_hour
endpoint_p90_month
endpoint_p95
endpoint_p95_day
endpoint_p95_hour
endpoint_p95_month
endpoint_p99
endpoint_p99_day
endpoint_p99_hour
endpoint_p99_month
endpoint_relation_cpm
endpoint_relation_cpm_day
endpoint_relation_cpm_hour
endpoint_relation_cpm_month
endpoint_relation_resp_time
endpoint_relation_resp_time_day
endpoint_relation_resp_time_hour
endpoint_relation_resp_time_month
endpoint_relation_server_side
endpoint_relation_server_side_day
endpoint_relation_server_side_hour
endpoint_relation_server_side_month
endpoint_sla
endpoint_sla_day
endpoint_sla_hour
endpoint_sla_month
envoy_heap_memory_max_used
envoy_heap_memory_max_used_day
envoy_heap_memory_max_used_hour
envoy_heap_memory_max_used_month
envoy_parent_connections_used
envoy_parent_connections_used_day
envoy_parent_connections_used_hour
envoy_parent_connections_used_month
envoy_total_connections_used
envoy_total_connections_used_day
envoy_total_connections_used_hour
envoy_total_connections_used_month
http_access_log
instance_clr_available_completion_port_threads
instance_clr_available_completion_port_threads_day
instance_clr_available_completion_port_threads_hour
instance_clr_available_completion_port_threads_month
instance_clr_available_worker_threads
instance_clr_available_worker_threads_day
instance_clr_available_worker_threads_hour
instance_clr_available_worker_threads_month
instance_clr_cpu
instance_clr_cpu_day
instance_clr_cpu_hour
instance_clr_cpu_month
instance_clr_gen0_collect_count
instance_clr_gen0_collect_count_day
instance_clr_gen0_collect_count_hour
instance_clr_gen0_collect_count_month
instance_clr_gen1_collect_count
instance_clr_gen1_collect_count_day
instance_clr_gen1_collect_count_hour
instance_clr_gen1_collect_count_month
instance_clr_gen2_collect_count
instance_clr_gen2_collect_count_day
instance_clr_gen2_collect_count_hour
instance_clr_gen2_collect_count_month
instance_clr_heap_memory
instance_clr_heap_memory_day
instance_clr_heap_memory_hour
instance_clr_heap_memory_month
instance_clr_max_completion_port_threads
instance_clr_max_completion_port_threads_day
instance_clr_max_completion_port_threads_hour
instance_clr_max_completion_port_threads_month
instance_clr_max_worker_threads
instance_clr_max_worker_threads_day
instance_clr_max_worker_threads_hour
instance_clr_max_worker_threads_month
instance_jvm_cpu
instance_jvm_cpu_day
instance_jvm_cpu_hour
instance_jvm_cpu_month
instance_jvm_memory_heap
instance_jvm_memory_heap_day
instance_jvm_memory_heap_hour
instance_jvm_memory_heap_max
instance_jvm_memory_heap_max_day
instance_jvm_memory_heap_max_hour
instance_jvm_memory_heap_max_month
instance_jvm_memory_heap_month
instance_jvm_memory_noheap
instance_jvm_memory_noheap_day
instance_jvm_memory_noheap_hour
instance_jvm_memory_noheap_max
instance_jvm_memory_noheap_max_day
instance_jvm_memory_noheap_max_hour
instance_jvm_memory_noheap_max_month
instance_jvm_memory_noheap_month
instance_jvm_old_gc_count
instance_jvm_old_gc_count_day
instance_jvm_old_gc_count_hour
instance_jvm_old_gc_count_month
instance_jvm_old_gc_time
instance_jvm_old_gc_time_day
instance_jvm_old_gc_time_hour
instance_jvm_old_gc_time_month
instance_jvm_young_gc_count
instance_jvm_young_gc_count_day
instance_jvm_young_gc_count_hour
instance_jvm_young_gc_count_month
instance_jvm_young_gc_time
instance_jvm_young_gc_time_day
instance_jvm_young_gc_time_hour
instance_jvm_young_gc_time_month
jaeger_span
segment
service_cpm
service_cpm_day
service_cpm_hour
service_cpm_month
service_instance_cpm
service_instance_cpm_day
service_instance_cpm_hour
service_instance_cpm_month
service_instance_resp_time
service_instance_resp_time_day
service_instance_resp_time_hour
service_instance_resp_time_month
service_instance_sla
service_instance_sla_day
service_instance_sla_hour
service_instance_sla_month
service_p50
service_p50_day
service_p50_hour
service_p50_month
service_p75
service_p75_day
service_p75_hour
service_p75_month
service_p90
service_p90_day
service_p90_hour
service_p90_month
service_p95
service_p95_day
service_p95_hour
service_p95_month
service_p99
service_p99_day
service_p99_hour
service_p99_month
service_relation_client_call_sla
service_relation_client_call_sla_day
service_relation_client_call_sla_hour
service_relation_client_call_sla_month
service_relation_client_cpm
service_relation_client_cpm_day
service_relation_client_cpm_hour
service_relation_client_cpm_month
service_relation_client_p50
service_relation_client_p50_day
service_relation_client_p50_hour
service_relation_client_p50_month
service_relation_client_p75
service_relation_client_p75_day
service_relation_client_p75_hour
service_relation_client_p75_month
service_relation_client_p90
service_relation_client_p90_day
service_relation_client_p90_hour
service_relation_client_p90_month
service_relation_client_p95
service_relation_client_p95_day
service_relation_client_p95_hour
service_relation_client_p95_month
service_relation_client_p99
service_relation_client_p99_day
service_relation_client_p99_hour
service_relation_client_p99_month
service_relation_client_resp_time
service_relation_client_resp_time_day
service_relation_client_resp_time_hour
service_relation_client_resp_time_month
service_relation_client_side
service_relation_client_side_day
service_relation_client_side_hour
service_relation_client_side_month
service_relation_server_call_sla
service_relation_server_call_sla_day
service_relation_server_call_sla_hour
service_relation_server_call_sla_month
service_relation_server_cpm
service_relation_server_cpm_day
service_relation_server_cpm_hour
service_relation_server_cpm_month
service_relation_server_p50
service_relation_server_p50_day
service_relation_server_p50_hour
service_relation_server_p50_month
service_relation_server_p75
service_relation_server_p75_day
service_relation_server_p75_hour
service_relation_server_p75_month
service_relation_server_p90
service_relation_server_p90_day
service_relation_server_p90_hour
service_relation_server_p90_month
service_relation_server_p95
service_relation_server_p95_day
service_relation_server_p95_hour
service_relation_server_p95_month
service_relation_server_p99
service_relation_server_p99_day
service_relation_server_p99_hour
service_relation_server_p99_month
service_relation_server_resp_time
service_relation_server_resp_time_day
service_relation_server_resp_time_hour
service_relation_server_resp_time_month
service_relation_server_side
service_relation_server_side_day
service_relation_server_side_hour
service_relation_server_side_month
service_resp_time
service_resp_time_day
service_resp_time_hour
service_resp_time_month
service_sla
service_sla_day
service_sla_hour
service_sla_month
top_n_database_statement
zipkin_span
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-589518601
@wu-sheng I agree with you, however sorry to say currently `user_*` can not be used on configuration of sharding proxy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-589509147
> Using a Regex or expression to represent the tables on configuration of sharding proxy?
Yes. I think that is making sense even not for SkyWalking. Such as in the CRM system, the user-related tables should be sharded through `user_id`, then a rule targeting `user_*` tables is much better than configure them one by one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-590197372
Maybe i gave a bad example. A better example should be table `user`, `order`, `product` and so on. 😅
Sorry that `user_*` is not supported so far, in this case what should we do next?
@wu-sheng
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] tristaZero commented on issue #4131: ShardingSphere
storage configuration, e2e test and doc for SkyWalking 7
Posted by GitBox <gi...@apache.org>.
tristaZero commented on issue #4131: ShardingSphere storage configuration, e2e test and doc for SkyWalking 7
URL: https://github.com/apache/skywalking/issues/4131#issuecomment-593278342
@wu-sheng Hi, I want to double-check the configuration `my.cnf` of ShardingProxy is supposed to locate in `root_dir/install/shardingsphere/proxy/` to provide user or e2e test, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services