You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "18835572909 (via GitHub)" <gi...@apache.org> on 2025/11/12 11:19:02 UTC
[I] Sharding5.3.2：shadow rule error [shardingsphere]
18835572909 opened a new issue, #37081:
URL: https://github.com/apache/shardingsphere/issues/37081
<img width="2945" height="1973" alt="Image" src="https://github.com/user-attachments/assets/f885d995-63c9-4ecf-aa50-a21eb07a8c7a" />
``` Version ```
Springboot 2.7.18
ShardingJdbc 5.3.2
``` Resource step ```
PartialSQLRouteExecutor#route
=> step1: sharding rule => RouteUnit(dataSourceMap<db_1,db_1>, tableMap<order_tlb,order_tlb_1>);
=> step2: shadow rule
=> ShadowSQLRouter#decorateRouteContext
=>AbstractShadowDMLStatementRouteEngine#route
=>ShadowRouteEngine#decorateRouteContext
``` yml ```
props:
sql-show: true
dataSources:
db_0:
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: jdbc:mysql://ip1:3306/db_0?characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8&allowMultiQueries=true&allowPublicKeyRetrieval=true
username: root
password: ***
db_1:
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: jdbc:mysql://ip1:3306/db_1?characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8&allowMultiQueries=true&allowPublicKeyRetrieval=true
username: root
password: ***
shadow_0:
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: jdbc:mysql://ip1:3306/shadow_0?characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8&allowMultiQueries=true&allowPublicKeyRetrieval=true
username: root
password: ***
shadow_1:
dataSourceClassName: com.zaxxer.hikari.HikariDataSource
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: jdbc:mysql://ip1:3306/shadow_1?characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8&allowMultiQueries=true&allowPublicKeyRetrieval=true
username: root
password: ***
rules:
- !SHARDING
shardingAlgorithms:
table_inline:
type: INLINE
props:
algorithm-expression: order_tbl_$->{Math.abs(order_no.hashCode()) % 2}
database_inline:
type: INLINE
props:
algorithm-expression: db_$->{Math.abs(user_id.hashCode()) % 2}
defaultDatabaseStrategy:
standard:
shardingColumn: user_id
shardingAlgorithmName: database_inline
defaultTableStrategy:
standard:
shardingColumn: order_no
shardingAlgorithmName: table_inline
tables:
order_tbl:
actualDataNodes: db_$->{0..1}.order_tbl_$->{0..1}
- !SHADOW
dataSources:
shadowDataSource1:
productionDataSourceName: db_1
shadowDataSourceName: shadow_1
tables:
order_tbl:
dataSourceNames:
- shadowDataSource1
shadowAlgorithmNames:
- user_id_value
shadowAlgorithms:
user_id_value:
type: VALUE_MATCH
props:
operation: select
column: user_id
value: x3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "terrymanu (via GitHub)" <gi...@apache.org>.
terrymanu commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3531208378
After carefully analyzing your configuration and problem description, I confirm this is a design flaw in the ShardingSphere framework when handling combined sharding + shadow rule scenarios.
Root Cause
The current data source mapping logic in the ShadowRouteEngine.decorateRouteContext() method has issues:
// Current problematic code logic
String actualName = each.getDataSourceMapper().getActualName();
String sourceDataSourceName = shadowRule.getSourceDataSourceName(actualName);
String shadowDataSourceName = shadowRule.getShadowDataSourceMappings().get(sourceDataSourceName);
The problem is that the actualName after sharding routing may have been modified by the framework, causing getSourceDataSourceName() to fail to correctly reverse-resolve to the original production data source name,
ultimately resulting in shadowDataSourceName being null.
Fix Design Solutions
I suggest adopting one of the following approaches to fix this issue:
Solution 1: Enhanced Data Source Name Tracking Mechanism
// Add original data source tracking in RouteUnit
public final class RouteUnit {
private final String originalDataSourceName; // Preserve original data source name
private final DataSourceMapper dataSourceMapper;
public RouteUnit(final String originalDataSourceName, final DataSourceMapper dataSourceMapper) {
this.originalDataSourceName = originalDataSourceName;
this.dataSourceMapper = dataSourceMapper;
}
public String getOriginalDataSourceName() {
return originalDataSourceName;
}
}
Solution 2: Improved Reverse Mapping Algorithm
// Enhance getSourceDataSourceName method in ShadowRule
public String getSourceDataSourceName(String routedDataSourceName) {
// Exact match
if (shadowDataSourceMappings.containsKey(routedDataSourceName)) {
return routedDataSourceName;
}
// Fuzzy matching for sharding scenarios
for (String dataSourceName : shadowDataSourceMappings.keySet()) {
if (routedDataSourceName.startsWith(dataSourceName) ||
routedDataSourceName.contains(dataSourceName)) {
return dataSourceName;
}
}
return null;
}
Solution 3: Use Route Context to Pass Mapping Relationships
// Maintain data source name mapping in RouteContext
public final class RouteContext {
private final Map<String, String> dataSourceNameMapping = new LinkedHashMap<>();
public void addDataSourceMapping(String original, String routed) {
dataSourceNameMapping.put(routed, original);
}
public String getOriginalDataSourceName(String routedName) {
return dataSourceNameMapping.getOrDefault(routedName, routedName);
}
}
Invitation to Participate
If you're interested in fixing this issue, we'd very much welcome you to submit a PR to complete this improvement! This is a great opportunity to contribute.
Suggested fix steps:
1. Choose one of the solutions above (I recommend Solution 1 as the clearest approach)
2. Implement the changes in your local branch
3. Write comprehensive unit tests covering various scenarios
4. Ensure existing test cases pass
5. Submit a PR and describe your fix approach
Your fix will help many developers using the combined sharding + shadow rules functionality. If you have any questions about implementation details, feel free to discuss in the PR, and we'll assist with code review and
merging.
Thank you for your contribution to the ShardingSphere project!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "18835572909 (via GitHub)" <gi...@apache.org>.
18835572909 commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3525354195
<img width="3022" height="1323" alt="Image" src="https://github.com/user-attachments/assets/2d1ef1f7-7fd7-4e22-ad32-03007b67b07b" />
This picture illustrates： isShadow
<img width="3487" height="1461" alt="Image" src="https://github.com/user-attachments/assets/b81bb557-db04-45bc-9831-922d080d6d1f" />
this is query log
<img width="1791" height="1048" alt="Image" src="https://github.com/user-attachments/assets/354d7b2c-00c5-406c-bc97-6410dbf0407a" />
<img width="1789" height="976" alt="Image" src="https://github.com/user-attachments/assets/3a8e6192-ef77-4840-8e58-402b1bbff39f" />
I understand： each.getDataSourceMapper().getActualName() is shading route value
but, shadowDataSourceMappings.get(actualDataSourceName); It must not be obtainable。 Is that so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "terrymanu (via GitHub)" <gi...@apache.org>.
terrymanu commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3528825331
Please use text only, can include codes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "18835572909 (via GitHub)" <gi...@apache.org>.
18835572909 commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3527778365
I have tried my best to provide it, so what should I do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "terrymanu (via GitHub)" <gi...@apache.org>.
terrymanu commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3524926709
I have carefully analyzed the issue you reported. According to your description, ShardingSphere 5.3.2 encounters an error at the ShadowSQLRouter#decorateRouteContext step when using a combination of sharding rules and
shadow rules.
Current Understanding
Your configuration includes:
- Data sources: db_0, db_1 (production) and shadow_0, shadow_1 (shadow)
- Sharding rules: Database sharding based on user_id, table sharding based on order_no
- Shadow rules: db_1 mapped to shadow_1, using VALUE_MATCH algorithm to match user_id='x3'
The sharding rule works correctly, but it fails at the shadow rule processing step.
More Information Needed
To accurately identify the root cause, please provide the following information:
1. Complete error stack trace
- Currently only showing method call chain, need specific exception information and complete stack trace
- Please include exception type and error message
2. Actual executed SQL statement and parameters
- Complete SQL statement
- Parameter binding values (especially the actual value of user_id)
3. Simplified reproduction code
- Minimal test code
- Remove unnecessary business logic
4. Log configuration
- Please confirm ShardingSphere SQL logging is enabled (you have configured sql-show: true)
- Provide complete log output during execution
Possible Cause Analysis
Based on code analysis, the issue might occur in:
1. Data source mapping logic
- ShadowSQLRouter needs to correctly identify whether the sharded data source is a shadow data source
- Check the return value of findProductionDataSourceName method
2. Shadow algorithm matching
- VALUE_MATCH algorithm needs to correctly obtain SQL parameters
- Check if user_id parameter passing is correct
3. Configuration consistency
- Ensure data source names in shadow rules match actual data source names produced by sharding rules
Recommended Check Steps
1. Verify shadow algorithm configuration
shadowAlgorithms:
user_id_value:
type: VALUE_MATCH
props:
operation: select # Ensure it matches actual SQL type
column: user_id
value: x3
2. Check data source mapping
- Confirm shadowDataSource1 configuration is correct
- Verify productionDataSourceName: db_1 is consistent with sharding results
3. Enable detailed logging
props:
sql-show: true
sql-simple: true
Please provide the above information, which will help us accurately identify the issue and provide a solution. If this is a ShardingSphere bug, we will fix it based on the specific situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "terrymanu (via GitHub)" <gi...@apache.org>.
terrymanu commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3527486157
Could you please avoid using screenshots?
LLM cannot directly read images, and relying on screenshots may result in incomplete or inaccurate extraction of the issue details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "terrymanu (via GitHub)" <gi...@apache.org>.
terrymanu commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3530739779
What is practiceName？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "18835572909 (via GitHub)" <gi...@apache.org>.
18835572909 commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3531105206
oh, I wrote it wrong. Is ： actualName
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "18835572909 (via GitHub)" <gi...@apache.org>.
18835572909 commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3525378825
this is my copy demo : https://github.com/18835572909/shadingjdbc-shadow-5.3.2.git
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "18835572909 (via GitHub)" <gi...@apache.org>.
18835572909 commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3530593345
org.apache.shardingsphere.shadow.route.engine.ShadowRouteEngine#decorateRouteContext
default void decorateRouteContext(final RouteContext routeContext, final ShadowRule shadowRule, final Map<String, String> shadowDataSourceMappings) {
Collection<RouteUnit> toBeRemovedRouteUnit = new LinkedList<>();
Collection<RouteUnit> toBeAddedRouteUnit = new LinkedList<>();
for (RouteUnit each : routeContext.getRouteUnits()) {
String logicName = each.getDataSourceMapper().getLogicName();
String actualName = each.getDataSourceMapper().getActualName();
Optional<String> sourceDataSourceName = shadowRule.getSourceDataSourceName(actualName);
if (sourceDataSourceName.isPresent()) {
String shadowDataSourceName = shadowDataSourceMappings.get(sourceDataSourceName.get());
toBeRemovedRouteUnit.add(each);
toBeAddedRouteUnit.add(null == shadowDataSourceName
? new RouteUnit(new RouteMapper(logicName, sourceDataSourceName.get()), each.getTableMappers())
: new RouteUnit(new RouteMapper(logicName, shadowDataSourceName), each.getTableMappers()));
}
}
routeContext.getRouteUnits().removeAll(toBeRemovedRouteUnit);
routeContext.getRouteUnits().addAll(toBeAddedRouteUnit);
}
This is the source code:
shadowRule.getSourceDataSourceName(actualName);//
I don't quite understand this part of him. Is the variable 'practiceName' the result of shardingRule？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Sharding5.3.2：shadow rule error [shardingsphere]
Posted by "18835572909 (via GitHub)" <gi...@apache.org>.
18835572909 commented on issue #37081:
URL: https://github.com/apache/shardingsphere/issues/37081#issuecomment-3532113924
Really appreciate your deep dive into this issue and the clear step-by-step breakdown.
I’ll follow your guidance: Prepare a PR branch (fix/shadow-rule-parse)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org