You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by z11373 <z1...@outlook.com> on 2017/02/14 16:06:32 UTC
exception from thrift
Hi,
I have c++ client talking to Accumulo via Thrift.
The client app is simply running in the loop, and inside the loop it creates
batch writer, add mutation with random length of string data, call that
batch writer flush() and close() methods. The code is running fine, except
when running it with multiple instances, and after some number of
iterations, some of the instances may start hitting this exception:
org.apache.accumulo.core.client.MutationsRejectedException: # constraint
violations : 0 security codes: {} # server errors 0 # exceptions 1
Do you know what causing this exception? Does it have anything to do with
the fact the client is accessing Accumulo via Thrift, or problem with the
Thrift proxy itself?
Thanks,
Z
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/exception-from-thrift-tp19919.html
Sent from the Developers mailing list archive at Nabble.com.
Re: exception from thrift
Posted by z11373 <z1...@outlook.com>.
Thanks Josh! You are right, I am using the Thrift proxy with the proxy server
came with Accumulo installation. Actually, now it's getting obvious that the
problem is likely not Thrift itself, but Accumulo side.
That exception was due to timeout (thanks to Mike Moss who helped me
pointing the issue), and looking at the code, I passed 10s timeout setting
to batch writer. That setting may not be long enough as this test client
(with 50 instances running) will write tenths of MBs of data, and when
Accumulo server is busy.
To proof that, I wrote a simple Java client program that mimic same thing
like that C++ client, and it also hit the same issue when I set 10s timeout.
I don't see the exception when I tried with 60s timeout.
I have side question though, for this case when timeout happens and write
failed, to retry the write I need to re-create the connection too, right?
Thanks,
Z
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/exception-from-thrift-tp19919p19944.html
Sent from the Developers mailing list archive at Nabble.com.
Re: exception from thrift
Posted by Josh Elser <jo...@gmail.com>.
I think he's using the Thrift proxy, Marc (given the first message).
Z -- I would check the Thrift proxy's log to see if the exception was
logged there, or the TabletServer(s) log otherwise. It might just be a
defect that the Thrift proxy server doesn't pass along that information
to your client correctly..
Marc P. wrote:
> Z,
> Is this your custom thrift RPC call or an object against the thrift
> proxy? If you are connecting to the proxy, then perhaps explore the
> exception further there or within Accumulo's Monitor. Depending on the
> problem that's occurring a retry will likely yield the same result.
> Do you have access to either to determine the problem? If you are using the
> proxy I would venture to guess that the rejection is likely
> one that isn't to be ignored.
>
> If you are doing your own thrift RPC calls, then I may be able to provide
> some code snippets to help as you get different
> information from writing your own calls; however, I don't want to assume
> you have gone down this route and provide incorrect information.
>
> On Tue, Feb 14, 2017 at 5:22 PM, z11373<z1...@outlook.com> wrote:
>
>> Thanks Marc for quick response.
>> The code throws exception when its calling flush(), and I was told since we
>> are using older version of Thrift C++ files, complete exception info may
>> not
>> be returned. Here is snippet of my code to catch it:
>>
>> try {
>> writer.addMutation(mutation);
>> writer.flush();
>> }
>> catch (const accumulo::AccumuloSecurityException& ase) {
>> std::cerr<< stage<< " Hit accumulo security exception: "<<
>> ase.msg<< std::endl;
>> }
>>
>> and "org.apache.accumulo.core.client.MutationsRejectedException: #
>> constraint violations : 0 security codes: {} # server errors 0 #
>> exceptions 1" is the only exception message I got.
>>
>>
>> I can write code to retry when hitting that exception, do I just retry
>> calling flush(), or I need to re-create the batch writer object (and add
>> mutation again)?
>>
>> Thanks,
>> Z
>>
>>
>>
>> --
>> View this message in context: http://apache-accumulo.1065345
>> .n5.nabble.com/exception-from-thrift-tp19919p19921.html
>> Sent from the Developers mailing list archive at Nabble.com.
>>
>
Re: exception from thrift
Posted by "Marc P." <ma...@gmail.com>.
Z,
Is this your custom thrift RPC call or an object against the thrift
proxy? If you are connecting to the proxy, then perhaps explore the
exception further there or within Accumulo's Monitor. Depending on the
problem that's occurring a retry will likely yield the same result.
Do you have access to either to determine the problem? If you are using the
proxy I would venture to guess that the rejection is likely
one that isn't to be ignored.
If you are doing your own thrift RPC calls, then I may be able to provide
some code snippets to help as you get different
information from writing your own calls; however, I don't want to assume
you have gone down this route and provide incorrect information.
On Tue, Feb 14, 2017 at 5:22 PM, z11373 <z1...@outlook.com> wrote:
> Thanks Marc for quick response.
> The code throws exception when its calling flush(), and I was told since we
> are using older version of Thrift C++ files, complete exception info may
> not
> be returned. Here is snippet of my code to catch it:
>
> try {
> writer.addMutation(mutation);
> writer.flush();
> }
> catch (const accumulo::AccumuloSecurityException& ase) {
> std::cerr << stage << " Hit accumulo security exception: " <<
> ase.msg << std::endl;
> }
>
> and "org.apache.accumulo.core.client.MutationsRejectedException: #
> constraint violations : 0 security codes: {} # server errors 0 #
> exceptions 1" is the only exception message I got.
>
>
> I can write code to retry when hitting that exception, do I just retry
> calling flush(), or I need to re-create the batch writer object (and add
> mutation again)?
>
> Thanks,
> Z
>
>
>
> --
> View this message in context: http://apache-accumulo.1065345
> .n5.nabble.com/exception-from-thrift-tp19919p19921.html
> Sent from the Developers mailing list archive at Nabble.com.
>
Re: exception from thrift
Posted by z11373 <z1...@outlook.com>.
Thanks Marc for quick response.
The code throws exception when its calling flush(), and I was told since we
are using older version of Thrift C++ files, complete exception info may not
be returned. Here is snippet of my code to catch it:
try {
writer.addMutation(mutation);
writer.flush();
}
catch (const accumulo::AccumuloSecurityException& ase) {
std::cerr << stage << " Hit accumulo security exception: " <<
ase.msg << std::endl;
}
and "org.apache.accumulo.core.client.MutationsRejectedException: #
constraint violations : 0 security codes: {} # server errors 0 #
exceptions 1" is the only exception message I got.
I can write code to retry when hitting that exception, do I just retry
calling flush(), or I need to re-create the batch writer object (and add
mutation again)?
Thanks,
Z
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/exception-from-thrift-tp19919p19921.html
Sent from the Developers mailing list archive at Nabble.com.
Re: exception from thrift
Posted by "Marc P." <ma...@gmail.com>.
Could be a number of causes. If you are not using the proxy I'd ensure the
tserver is still hosting after a split. If you are you can get more info
from the exception.
In my CPP client I take failed mutations and requeue them depending on the
exception details. If the extent locations have changed I update. If a
security issue then I return an error.
On Tue, Feb 14, 2017, 11:11 AM z11373 <z1...@outlook.com> wrote:
> Hi,
> I have c++ client talking to Accumulo via Thrift.
> The client app is simply running in the loop, and inside the loop it
> creates
> batch writer, add mutation with random length of string data, call that
> batch writer flush() and close() methods. The code is running fine, except
> when running it with multiple instances, and after some number of
> iterations, some of the instances may start hitting this exception:
>
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint
> violations : 0 security codes: {} # server errors 0 # exceptions 1
>
> Do you know what causing this exception? Does it have anything to do with
> the fact the client is accessing Accumulo via Thrift, or problem with the
> Thrift proxy itself?
>
>
> Thanks,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/exception-from-thrift-tp19919.html
> Sent from the Developers mailing list archive at Nabble.com.
>