You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/05/31 13:53:48 UTC
[GitHub] [druid] rickymarini opened a new issue, #12586: Metadata storage insert flaw when mysql/mariadb
rickymarini opened a new issue, #12586:
URL: https://github.com/apache/druid/issues/12586
Hi **Team**,
I was testing [proxySQL](https://github.com/sysown/proxysql) on our site and I found a flaw in how druid tries to write segments on the metadata storage DB, when mysql/mariadb is in use.
Mariadb JDBC has an [option](https://mariadb.com/kb/en/about-mariadb-connector-j/#:~:text=rewriteBatchedStatements) to force statements rewrite in case of batching:
```rewriteBatchedStatements=true```
If I add this option to the connection string, driver seems to do its job sending out multi rows INSERT but then druid is not able to publish new segments, as the validation check seems to be expecting EXACTLY **a single row** for each statement executed (or at least, this is my understanding), while the driver option concatenates several INSERT into one, leading to an increased (>=1) *affectedRows*
```java
final int[] affectedRows = preparedBatch.execute();
final boolean succeeded = Arrays.stream(affectedRows).allMatch(eachAffectedRows -> eachAffectedRows == 1);
if (succeeded) {
log.infoSegments(partition, "Published segments to DB");
} else {
final List<DataSegment> failedToPublish = IntStream.range(0, partition.size())
.filter(i -> affectedRows[i] != 1)
.mapToObj(partition::get)
.collect(Collectors.toList());
throw new ISE(
"Failed to publish segments to DB: %s",
SegmentUtils.commaSeparatedIdentifiers(failedToPublish)
);
```
```
overlord-2022-05-31-9.log:Caused by: org.skife.jdbi.v2.exceptions.CallbackFailedException: org.apache.druid.java.util.common.ISE: Failed to publish segments to DB: [xyz, xzy]
```
I understand and respect this, since it is probably to maximize data integrity when shooting new segments in the segments table.
My proposal, if this is confirmed, is to fix this logic in case we have the driver option turned on, or in case it is too risky or cumbersome to do that, to mention this JDBC parameter as **not supported** in the metadata storage doc.
Thanks for your review/opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Metadata storage insert flaw when mysql/mariadb (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12586:
URL: https://github.com/apache/druid/issues/12586#issuecomment-1859342156
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Metadata storage insert flaw when mysql/mariadb (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12586: Metadata storage insert flaw when mysql/mariadb
URL: https://github.com/apache/druid/issues/12586
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Metadata storage insert flaw when mysql/mariadb (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12586:
URL: https://github.com/apache/druid/issues/12586#issuecomment-1892900305
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org