You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Sanooj Padmakumar <p....@gmail.com> on 2016/08/24 05:56:30 UTC
RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time
Hi All,
I get this error one one of the nodes where we have an application running.
It comes only after a certain duration and once the application is
restarted things will start working normally..Any inputs as to why this
might be happening will be of great help.
org.springframework.jdbc.UncategorizedSQLException:
PreparedStatementCallback; uncategorized SQLException for SQL [<<query here
>>> ]; SQL state [null]; error code [0];
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
1 action: IOException: 1 time, ; nested exception is
org.apache.phoenix.execute.CommitException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
1 action: IOException: 1 time,
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:660)
at
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:909)
at
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:933)
at
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:313)
at Caused by: org.apache.phoenix.execute.CommitException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
1 action: IOException: 1 time,
at
org.apache.phoenix.execute.MutationState.commit(MutationState.java:473)
at
org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:472)
at
org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:469)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.java:469)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:323)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:312)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:310)
at
org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeUpdate(PhoenixPreparedStatement.java:200)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:916)
at
org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:909)
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:644)
... 11 more
Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
1 action: IOException: 1 time,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1568)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1003)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1017)
at
org.apache.phoenix.execute.MutationState.commit(MutationState.java:444)
... 25 more
--
Thanks,
Sanooj Padmakumar
Re: RetriesExhaustedWithDetailsException: Failed 1 action:
IOException: 1 time
Posted by Ankit Singhal <an...@gmail.com>.
can you check whether your hbase is stable or not (you can use hbck tool to
see any inconsistencies).
On Sat, Aug 27, 2016 at 10:41 PM, Sanooj Padmakumar <p....@gmail.com>
wrote:
> Hi All,
>
> I am getting the same exception , this time when running a Phoenix MR (
> https://phoenix.apache.org/phoenix_mr.html) .. The MR works just fine if
> I am doing a select with limit to some 10 rows.. but when I do the same
> with a lot of data, I start getting the below exception after 66% of
> reducer..
>
> 6/08/27 10:04:35 INFO mapreduce.Job: Task Id :
> attempt_1471862728027_0103_r_000036_0, Status : FAILED
> Error: java.lang.RuntimeException: org.apache.phoenix.execute.CommitException:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 2000 actions: IOException: 2000 times,
> at org.apache.phoenix.mapreduce.PhoenixRecordWriter.close(
> PhoenixRecordWriter.java:62)
> at org.apache.hadoop.mapred.ReduceTask$
> NewTrackingRecordWriter.close(ReduceTask.java:550)
> ............
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 2000 actions: IOException: 2000 times,
> at org.apache.hadoop.hbase.client.AsyncProcess$
> BatchErrors.makeException(AsyncProcess.java:227)
> at org.apache.hadoop.hbase.client.AsyncProcess$
> BatchErrors.access$1700(AsyncProcess.java:207)
> at org.apache.hadoop.hbase.client.AsyncProcess$
> AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1568)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1003)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1017)
> at org.apache.phoenix.execute.MutationState.commit(
> MutationState.java:444)
> ... 13 more
>
> When I look at the failed reducers log I am seeing a lot of logs like
> these..
>
> 2016-08-27 10:07:05,834 ERROR [main] org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0 location for {"totalColumns":25,"families":{"p":[{"timestamp":1472317624237,"tag":[],"qualifier"
>
> But as I said with very less number of rows in the select query, the MR works just fine and data is populated alright ? Any Phoenix/Hbase parameters
>
> that is should look into ?
>
> Thanks
>
> Sanooj Padmakumar
>
>
>
>
>
> On Wed, Aug 24, 2016 at 11:26 AM, Sanooj Padmakumar <p....@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I get this error one one of the nodes where we have an application
>> running. It comes only after a certain duration and once the application is
>> restarted things will start working normally..Any inputs as to why this
>> might be happening will be of great help.
>>
>>
>> org.springframework.jdbc.UncategorizedSQLException:
>> PreparedStatementCallback; uncategorized SQLException for SQL [<<query here
>> >>> ]; SQL state [null]; error code [0]; org.apache.hadoop.hbase.client
>> .RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1
>> time, ; nested exception is org.apache.phoenix.execute.CommitException:
>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> Failed 1 action: IOException: 1 time,
>> at org.springframework.jdbc.support.AbstractFallbackSQLExceptio
>> nTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
>> at org.springframework.jdbc.support.AbstractFallbackSQLExceptio
>> nTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
>> at org.springframework.jdbc.support.AbstractFallbackSQLExceptio
>> nTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
>> at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTempl
>> ate.java:660)
>> at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTempla
>> te.java:909)
>> at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTempla
>> te.java:933)
>> at org.springframework.jdbc.core.namedparam.NamedParameterJdbcT
>> emplate.update(NamedParameterJdbcTemplate.java:313)
>> at Caused by: org.apache.phoenix.execute.CommitException:
>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> Failed 1 action: IOException: 1 time,
>> at org.apache.phoenix.execute.MutationState.commit(MutationStat
>> e.java:473)
>> at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConn
>> ection.java:472)
>> at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConn
>> ection.java:469)
>> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>> at org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConn
>> ection.java:469)
>> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixState
>> ment.java:323)
>> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixState
>> ment.java:312)
>> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>> at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(Pho
>> enixStatement.java:310)
>> at org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeUpda
>> te(PhoenixPreparedStatement.java:200)
>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeU
>> pdate(DelegatingPreparedStatement.java:105)
>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeU
>> pdate(DelegatingPreparedStatement.java:105)
>> at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedSta
>> tement(JdbcTemplate.java:916)
>> at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedSta
>> tement(JdbcTemplate.java:909)
>> at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTempl
>> ate.java:644)
>> ... 11 more
>> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> Failed 1 action: IOException: 1 time,
>> at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.
>> makeException(AsyncProcess.java:227)
>> at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.
>> access$1700(AsyncProcess.java:207)
>> at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>> reImpl.getErrors(AsyncProcess.java:1568)
>> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1003)
>> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1017)
>> at org.apache.phoenix.execute.MutationState.commit(MutationStat
>> e.java:444)
>> ... 25 more
>>
>>
>> --
>> Thanks,
>> Sanooj Padmakumar
>>
>>
>
>
> --
> Thanks,
> Sanooj Padmakumar
>
Re: RetriesExhaustedWithDetailsException: Failed 1 action:
IOException: 1 time
Posted by Sanooj Padmakumar <p....@gmail.com>.
Hi All,
I am getting the same exception , this time when running a Phoenix MR (
https://phoenix.apache.org/phoenix_mr.html) .. The MR works just fine if I
am doing a select with limit to some 10 rows.. but when I do the same with
a lot of data, I start getting the below exception after 66% of reducer..
6/08/27 10:04:35 INFO mapreduce.Job: Task Id :
attempt_1471862728027_0103_r_000036_0, Status : FAILED
Error: java.lang.RuntimeException:
org.apache.phoenix.execute.CommitException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
2000 actions: IOException: 2000 times,
at
org.apache.phoenix.mapreduce.PhoenixRecordWriter.close(PhoenixRecordWriter.java:62)
at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:550)
............
Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
2000 actions: IOException: 2000 times,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207)
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1568)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1003)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1017)
at
org.apache.phoenix.execute.MutationState.commit(MutationState.java:444)
... 13 more
When I look at the failed reducers log I am seeing a lot of logs like
these..
2016-08-27 10:07:05,834 ERROR [main]
org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0
location for {"totalColumns":25,"families":{"p":[{"timestamp":1472317624237,"tag":[],"qualifier"
But as I said with very less number of rows in the select query, the
MR works just fine and data is populated alright ? Any Phoenix/Hbase
parameters
that is should look into ?
Thanks
Sanooj Padmakumar
On Wed, Aug 24, 2016 at 11:26 AM, Sanooj Padmakumar <p....@gmail.com>
wrote:
> Hi All,
>
> I get this error one one of the nodes where we have an application
> running. It comes only after a certain duration and once the application is
> restarted things will start working normally..Any inputs as to why this
> might be happening will be of great help.
>
>
> org.springframework.jdbc.UncategorizedSQLException:
> PreparedStatementCallback; uncategorized SQLException for SQL [<<query here
> >>> ]; SQL state [null]; error code [0]; org.apache.hadoop.hbase.client.
> RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1
> time, ; nested exception is org.apache.phoenix.execute.CommitException:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 1 action: IOException: 1 time,
> at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTr
> anslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
> at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTr
> anslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
> at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTr
> anslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
> at org.springframework.jdbc.core.JdbcTemplate.execute(
> JdbcTemplate.java:660)
> at org.springframework.jdbc.core.JdbcTemplate.update(
> JdbcTemplate.java:909)
> at org.springframework.jdbc.core.JdbcTemplate.update(
> JdbcTemplate.java:933)
> at org.springframework.jdbc.core.namedparam.
> NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:313)
> at Caused by: org.apache.phoenix.execute.CommitException:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 1 action: IOException: 1 time,
> at org.apache.phoenix.execute.MutationState.commit(
> MutationState.java:473)
> at org.apache.phoenix.jdbc.PhoenixConnection$3.call(
> PhoenixConnection.java:472)
> at org.apache.phoenix.jdbc.PhoenixConnection$3.call(
> PhoenixConnection.java:469)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at org.apache.phoenix.jdbc.PhoenixConnection.commit(
> PhoenixConnection.java:469)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(
> PhoenixStatement.java:323)
> at org.apache.phoenix.jdbc.PhoenixStatement$2.call(
> PhoenixStatement.java:312)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(
> PhoenixStatement.java:310)
> at org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeUpdate(
> PhoenixPreparedStatement.java:200)
> at org.apache.commons.dbcp.DelegatingPreparedStatement.
> executeUpdate(DelegatingPreparedStatement.java:105)
> at org.apache.commons.dbcp.DelegatingPreparedStatement.
> executeUpdate(DelegatingPreparedStatement.java:105)
> at org.springframework.jdbc.core.JdbcTemplate$2.
> doInPreparedStatement(JdbcTemplate.java:916)
> at org.springframework.jdbc.core.JdbcTemplate$2.
> doInPreparedStatement(JdbcTemplate.java:909)
> at org.springframework.jdbc.core.JdbcTemplate.execute(
> JdbcTemplate.java:644)
> ... 11 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 1 action: IOException: 1 time,
> at org.apache.hadoop.hbase.client.AsyncProcess$
> BatchErrors.makeException(AsyncProcess.java:227)
> at org.apache.hadoop.hbase.client.AsyncProcess$
> BatchErrors.access$1700(AsyncProcess.java:207)
> at org.apache.hadoop.hbase.client.AsyncProcess$
> AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1568)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1003)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1017)
> at org.apache.phoenix.execute.MutationState.commit(
> MutationState.java:444)
> ... 25 more
>
>
> --
> Thanks,
> Sanooj Padmakumar
>
>
--
Thanks,
Sanooj Padmakumar