You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by lihongliustc <gi...@git.apache.org> on 2016/06/27 14:41:03 UTC
[GitHub] spark pull request #13925: [SPARK-16226][SQL]change the way of JDBC commit
GitHub user lihongliustc opened a pull request:
https://github.com/apache/spark/pull/13925
[SPARK-16226][SQL]change the way of JDBC commit
## What changes were proposed in this pull request?
I change the way of commit.In the original version,it commit once in a partition.I commit after executeBatch.
## How was this patch tested?
unit tests
change the way of JDBC commit
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lihongliustc/spark patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13925.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13925
----
commit 133d410e50ee5d2e113b7739d66f8a47ee8cbe42
Author: lihongliustc <77...@qq.com>
Date: 2016-06-27T14:35:04Z
change the way of JDBC commit
change the way of JDBC commit
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13925
No, an exception will now only roll back anything that happened since the last commit. It is not the same. The whole reason it's 'blocking' is that you are holding open a transaction, but, that's the point.
There's an interesting point here though; this code is willing to proceed if the RDBMS does not support transactions at all, in which case it _will_ write partial results that can't be rolled back. Really I'd think that in this case you want to call `executeBatch()` only once at the end. I wonder if that's actually a bug (CC @rxin who wrote that bit?)
However, if that is acceptable, then perhaps a weaker JDBC transaction isolation level is acceptable for the case where transactions are supported. If you're willing to accept dirty reads with `TRANSACTION_READ_UNCOMMITTED`, then I don't think (?) the table has to be locked, which may solve your problem. It still means the writes would be rolled back in case of a failure, which is much of the point here according to comments.
CC @CK50 for the transaction question. Also barely related but shouldn't it just call `supportsTransactions()` at the beginning?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by lihongliustc <gi...@git.apache.org>.
Github user lihongliustc commented on the issue:
https://github.com/apache/spark/pull/13925
@srowen Thanks.I will close the PR.And I will try my best to do something like @gatorsmile said.Thanks to every participant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/13925
As @srowen and @hvanhovell said, changing commits is not right or not enough.
Even if the atomicity might not be needed in some cases, you need to implement something like `NOT ATOMIC CONTINUE ON SQLEXCEPTION`. However, that requires much more efforts to implement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by lihongliustc <gi...@git.apache.org>.
Github user lihongliustc commented on the issue:
https://github.com/apache/spark/pull/13925
@hvanhovell and @srowen I just change the way of commits.When there is an exception,it will rollback as usual.The "commited" is a flag of the method.Any exception in the try clause will cause the rollback of the partition.It is same as the situation before i changed the commits.It is still atomic .And I am sure that if we do not change commits,it will cause the blocking.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13925
Yes @maver1ck I think that's probably a good change, though it may end up being separate, and related to the isolation level change.
@lihongliustc this PR can be closed. The right solution for this, if any, is to reduce the transaction isolation level. That seems reasonable to me.
As to the issue of non-transactional RDBMSes seeing partial writes that can't be rolled back -- the only thing I can see as a solution is to not use batching, but I think that would run out of memory easily. It may just be how it is without transactions. The javadoc kind of says as much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by maver1ck <gi...@git.apache.org>.
Github user maver1ck commented on the issue:
https://github.com/apache/spark/pull/13925
@srowen
Recently I modified MySQL JDBC driver because both supportsDataManipulationTransactionsOnly() and supportsDataDefinitionAndDataManipulationTransactions() return false.
So maybe we can change this condition ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by lihongliustc <gi...@git.apache.org>.
Github user lihongliustc commented on the issue:
https://github.com/apache/spark/pull/13925
@gatorsmile Thanks.But I think it is still atomic.As I comment above.How do think about it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/13925
@lihongliustc could you take a look at the JIRA ticket, and respond to @srowen's very valid comment? I don't think we should do this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by lihongliustc <gi...@git.apache.org>.
Github user lihongliustc closed the pull request at:
https://github.com/apache/spark/pull/13925
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13925
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by CK50 <gi...@git.apache.org>.
Github user CK50 commented on the issue:
https://github.com/apache/spark/pull/13925
@sowen: yes, supportsTransactions() sounds good. I do not remember why I
took the other two conditions and whether this makes any difference.
I agree that committing in between defeats the purpose of the
transaction action handling.
On 28.06.2016 15:04, Sean Owen wrote:
>
> No, an exception will now only roll back anything that happened since
> the last commit. It is not the same. The whole reason it's 'blocking'
> is that you are holding open a transaction, but, that's the point.
>
> There's an interesting point here though; this code is willing to
> proceed if the RDBMS does not support transactions at all, in which
> case it /will/ write partial results that can't be rolled back. Really
> I'd think that in this case you want to call |executeBatch()| only
> once at the end. I wonder if that's actually a bug (CC @rxin
> <https://github.com/rxin> who wrote that bit?)
>
> However, if that is acceptable, then perhaps a weaker JDBC transaction
> isolation level is acceptable for the case where transactions are
> supported. If you're willing to accept dirty reads with
> |TRANSACTION_READ_UNCOMMITTED|, then I don't think (?) the table has
> to be locked, which may solve your problem. It still means the writes
> would be rolled back in case of a failure, which is much of the point
> here according to comments.
>
> CC @CK50 <https://github.com/CK50> for the transaction question. Also
> barely related but shouldn't it just call |supportsTransactions()| at
> the beginning?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/13925#issuecomment-229042189>,
> or mute the thread
> <https://github.com/notifications/unsubscribe/APQTSY-IpRC38pooh6dQGmRBT90Rgi7uks5qQRvTgaJpZM4I_Kd7>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by maver1ck <gi...@git.apache.org>.
Github user maver1ck commented on the issue:
https://github.com/apache/spark/pull/13925
@srowen
Maybe we should change this condition to `conn.getMetaData().supportsTransactions()` ?
I can prepare PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13925: [SPARK-16226][SQL]change the way of JDBC commit
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13925
I think the important change regards the transaction isolation level that's in effect here, but yes that change is also a good one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org