You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "PeiMouRen (via GitHub)" <gi...@apache.org> on 2023/06/14 10:36:00 UTC
[GitHub] [shardingsphere] PeiMouRen opened a new issue, #26342: [proxy5.3.2] NullPointerException occurred during batch insert
PeiMouRen opened a new issue, #26342:
URL: https://github.com/apache/shardingsphere/issues/26342
### Which version of ShardingSphere did you use?
5.3.2
### Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?
ShardingSphere-Prox
### Description
Using the kettle to import data to the proxy, when the batch number is 10, it's ok, but when the batch number is 500 or more, a NullPointerException occurred.
### Expected behavior
successfully import data
### Actual behavior
exception occurred
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1605910232
> > Hello, I don't know why, I now put the table in a custom schema, after I restart the proxy several times, now I can use the kettle to batch insert, batch size can be set to 500 or more, that is so strange!
>
> Did you execute DDL directly on Database instead of Proxy?
Yes, I executed ddl directly on the pg database, like `drop|create scheme xxx, create table xxx, create view xxx`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1594217070
@TeslaCN Of course, I've set the a attribute to true, and when the kettle batch number is 500, the proxy log looks like this:
As you can see, there is no sql in the log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1594268396
Yes, I have set this parameter to `true`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "TeslaCN (via GitHub)" <gi...@apache.org>.
TeslaCN commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1592243348
Could you enable sql-show in server.yaml and provide us logs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "TeslaCN (via GitHub)" <gi...@apache.org>.
TeslaCN commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1594269282
Could you try setting it `false`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "TeslaCN (via GitHub)" <gi...@apache.org>.
TeslaCN commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1605911031
> > > Hello, I don't know why, I now put the table in a custom schema, after I restart the proxy several times, now I can use the kettle to batch insert, batch size can be set to 500 or more, that is so strange!
> >
> >
> > Did you execute DDL directly on Database instead of Proxy?
>
> Yes, I executed ddl directly on the pg database, like `drop|create scheme xxx, create table xxx, create view xxx`
DDL must be executed through Proxy or executing DistSQL `REFRESH TABLE METADATA` after executed DDL.
https://shardingsphere.apache.org/document/current/en/user-manual/shardingsphere-proxy/distsql/syntax/ral/refresh-table-metadata/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "TeslaCN (via GitHub)" <gi...@apache.org>.
TeslaCN commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1605906695
> Hello, I don't know why, I now put the table in a custom schema, after I restart the proxy several times, now I can use the kettle to batch insert, batch size can be set to 500 or more, that is so strange!
Did you execute DDL directly on Database instead of Proxy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "TeslaCN (via GitHub)" <gi...@apache.org>.
TeslaCN commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1594385699
Could you try using schema `public`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1590942401
sharding rule:
```
rules:
- !SHARDING
tables:
T_Service_Order:
actualDataNodes: ds_${0..7}.T_Service_Order
databaseStrategy:
standard:
shardingColumn: Super_Account_No
shardingAlgorithmName: database_mod
shardingAlgorithms:
database_mod:
type: MOD
props:
sharding-count: 8
```
datasource:
```
dataSources:
ds_0:
url: jdbc:postgresql://127.0.0.1:5432/db0
username: test
password: 123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
ds_1:
url: jdbc:postgresql://127.0.0.1:5432/db1
username: test
password: 123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
ds_2:
url: jdbc:postgresql://127.0.0.1:5432/db2
username: test
password: 123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
ds_3:
url: jdbc:postgresql://127.0.0.1:5432/db3
username: test
password: 123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
ds_4:
url: jdbc:postgresql://127.0.0.1:5432/db4
username: test
password: 123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
ds_5:
url: jdbc:postgresql://127.0.0.1:5432/db5
username: test
password: 123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
ds_6:
url: jdbc:postgresql://127.0.0.1:5432/db6
username: test
password: 123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
ds_7:
url: jdbc:postgresql://127.0.0.1:5432/db7
username: test
password: 123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1594223497
When the number of kettle batches is 250, the proxy runs normally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1594271807
> Could you try setting it `false`?
ok, I'll try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "TeslaCN (via GitHub)" <gi...@apache.org>.
TeslaCN commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1605826859
Could you help us reproduce this issue? Such as providing DDL and your steps...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1605906488
Hello, I don't know why, I now put the table in a custom schema, after I restart the proxy several times, now I can use the kettle to batch insert, batch size can be set to 500 or more, that is so strange!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "TeslaCN (via GitHub)" <gi...@apache.org>.
TeslaCN commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1594246888
Is there any configuration about PostgreSQL JDBC `reWriteBatchedInserts` in kettle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1594278129
When I set the `reWriteBatchedInserts` attribute to `false`, it doesn't even support 250 batches. The exception message is the same as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1605911787
ok, I got it, thranks for your reply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen closed issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen closed issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
URL: https://github.com/apache/shardingsphere/issues/26342
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1594509531
yes, tables in `public` shema are fine, in other schema will report exception. Does proxy support for pg's schema not work so well now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] PeiMouRen commented on issue #26342: [proxy5.3.2] NullPointerException occurred during batch insert
Posted by "PeiMouRen (via GitHub)" <gi...@apache.org>.
PeiMouRen commented on issue #26342:
URL: https://github.com/apache/shardingsphere/issues/26342#issuecomment-1597992693
> yes, tables in `public` shema are fine, in other schema will report exception. Does proxy support for pg's schema not work so well now?
@TeslaCN
I reverse this conclusion, and when the table(table is `public` schema) has 10 or so columns, batch inserts are still limited to 250 or less.
Can you help me solve this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org