You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by William Shen <wi...@marinsoftware.com> on 2018/09/19 21:15:04 UTC
Encountering IllegalStateException while querying Phoenix
Hi there,
I have encountered the following exception while trying to query from
Phoenix (was able to generate the exception doing a simple SELECT
count(1)). I have verified (MD5) that each region server has the correct
phoenix jars. Would appreciate any guidance on how to proceed further in
troubleshooting this (or what could've caused this). Thank you!
java.lang.IllegalStateException: Expected single, aggregated KeyValue from
coprocessor, but instead received
keyvalues={\x03\x80\x00\x00\x00\x00\x8D\xB8Y\x80\x00\x00\x00\x01c$\xE7\x00\x04\x80\x00\x00\x00\x01\x0C\x95N\x80\x00\x00\x00\x01\xCCU\xF1/SL:_0/1525817954352/Put/vlen=1/seqid=0/value=x}
. Ensure aggregating coprocessors are loaded correctly on server
at org.apache.phoenix.util.TupleUtil.getAggregateValue(TupleUtil.java:88)
at
org.apache.phoenix.expression.aggregator.ClientAggregators.aggregate(ClientAggregators.java:54)
at
org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:74)
at
org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:779)
at sqlline.BufferedRows.<init>(BufferedRows.java:37)
at sqlline.SqlLine.print(SqlLine.java:1660)
at sqlline.Commands.execute(Commands.java:833)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:813)
at sqlline.SqlLine.begin(SqlLine.java:686)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:291)
- Will
Re: Encountering IllegalStateException while querying Phoenix
Posted by Jaanai Zhang <cl...@gmail.com>.
Are you sure you had restarted RS process? you can check
"phoenix-server.jar" whether exists in the classpath of HBase by "jinfo"
command
----------------------------------------
Jaanai Zhang
Best regards!
William Shen <wi...@marinsoftware.com> 于2018年9月20日周四 上午6:01写道：
> For anyone else interested: we ended up identifying one of the RS actually
> failed to load the UngroupedAggregateRegionObserver because of a strange
> XML parsing issue that was not occurring prior to this incident and not
> happening on the any other RS.
>
> Failed to load coprocessor org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver
> java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: jar:file:/opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/jars/hadoop-common-2.6.0-cdh5.9.2.jar!/core-default.xml; lineNumber: 196; columnNumber: 47; The string "--" is not permitted within comments.
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2656)
> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
> at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
> at org.apache.phoenix.util.PropertiesUtil.cloneConfig(PropertiesUtil.java:81)
> at org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.start(UngroupedAggregateRegionObserver.java:219)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:414)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:255)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:208)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:364)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:226)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:723)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:631)
> at sun.reflect.GeneratedConstructorAccessor23.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6145)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6449)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6421)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6377)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6328)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.xml.sax.SAXParseException; systemId: jar:file:/opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/jars/hadoop-common-2.6.0-cdh5.9.2.jar!/core-default.xml; lineNumber: 196; columnNumber: 47; The string "--" is not permitted within comments.
> at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
> at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2491)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2479)
> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2550)
> ... 27 more
>
>
>
> On Wed, Sep 19, 2018 at 2:15 PM William Shen <wi...@marinsoftware.com>
> wrote:
>
>> Hi there,
>>
>> I have encountered the following exception while trying to query from
>> Phoenix (was able to generate the exception doing a simple SELECT
>> count(1)). I have verified (MD5) that each region server has the correct
>> phoenix jars. Would appreciate any guidance on how to proceed further in
>> troubleshooting this (or what could've caused this). Thank you!
>>
>> java.lang.IllegalStateException: Expected single, aggregated KeyValue
>> from coprocessor, but instead received
>> keyvalues={\x03\x80\x00\x00\x00\x00\x8D\xB8Y\x80\x00\x00\x00\x01c$\xE7\x00\x04\x80\x00\x00\x00\x01\x0C\x95N\x80\x00\x00\x00\x01\xCCU\xF1/SL:_0/1525817954352/Put/vlen=1/seqid=0/value=x}
>>
>> . Ensure aggregating coprocessors are loaded correctly on server
>>
>> at org.apache.phoenix.util.TupleUtil.getAggregateValue(TupleUtil.java:88)
>>
>> at
>> org.apache.phoenix.expression.aggregator.ClientAggregators.aggregate(ClientAggregators.java:54)
>>
>> at
>> org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:74)
>>
>> at
>> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
>>
>> at
>> org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:779)
>>
>> at sqlline.BufferedRows.<init>(BufferedRows.java:37)
>>
>> at sqlline.SqlLine.print(SqlLine.java:1660)
>>
>> at sqlline.Commands.execute(Commands.java:833)
>>
>> at sqlline.Commands.sql(Commands.java:732)
>>
>> at sqlline.SqlLine.dispatch(SqlLine.java:813)
>>
>> at sqlline.SqlLine.begin(SqlLine.java:686)
>>
>> at sqlline.SqlLine.start(SqlLine.java:398)
>>
>> at sqlline.SqlLine.main(SqlLine.java:291)
>>
>> - Will
>>
>
Re: Encountering IllegalStateException while querying Phoenix
Posted by William Shen <wi...@marinsoftware.com>.
For anyone else interested: we ended up identifying one of the RS actually
failed to load the UngroupedAggregateRegionObserver because of a strange
XML parsing issue that was not occurring prior to this incident and not
happening on the any other RS.
Failed to load coprocessor
org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver
java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId:
jar:file:/opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/jars/hadoop-common-2.6.0-cdh5.9.2.jar!/core-default.xml;
lineNumber: 196; columnNumber: 47; The string "--" is not permitted
within comments.
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2656)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
at org.apache.phoenix.util.PropertiesUtil.cloneConfig(PropertiesUtil.java:81)
at org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.start(UngroupedAggregateRegionObserver.java:219)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:414)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:255)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:208)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:364)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:226)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:723)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:631)
at sun.reflect.GeneratedConstructorAccessor23.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6145)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6449)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6421)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6377)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6328)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.xml.sax.SAXParseException; systemId:
jar:file:/opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/jars/hadoop-common-2.6.0-cdh5.9.2.jar!/core-default.xml;
lineNumber: 196; columnNumber: 47; The string "--" is not permitted
within comments.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2491)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2479)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2550)
... 27 more
On Wed, Sep 19, 2018 at 2:15 PM William Shen <wi...@marinsoftware.com>
wrote:
> Hi there,
>
> I have encountered the following exception while trying to query from
> Phoenix (was able to generate the exception doing a simple SELECT
> count(1)). I have verified (MD5) that each region server has the correct
> phoenix jars. Would appreciate any guidance on how to proceed further in
> troubleshooting this (or what could've caused this). Thank you!
>
> java.lang.IllegalStateException: Expected single, aggregated KeyValue from
> coprocessor, but instead received
> keyvalues={\x03\x80\x00\x00\x00\x00\x8D\xB8Y\x80\x00\x00\x00\x01c$\xE7\x00\x04\x80\x00\x00\x00\x01\x0C\x95N\x80\x00\x00\x00\x01\xCCU\xF1/SL:_0/1525817954352/Put/vlen=1/seqid=0/value=x}
>
> . Ensure aggregating coprocessors are loaded correctly on server
>
> at org.apache.phoenix.util.TupleUtil.getAggregateValue(TupleUtil.java:88)
>
> at
> org.apache.phoenix.expression.aggregator.ClientAggregators.aggregate(ClientAggregators.java:54)
>
> at
> org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:74)
>
> at
> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
>
> at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:779)
>
> at sqlline.BufferedRows.<init>(BufferedRows.java:37)
>
> at sqlline.SqlLine.print(SqlLine.java:1660)
>
> at sqlline.Commands.execute(Commands.java:833)
>
> at sqlline.Commands.sql(Commands.java:732)
>
> at sqlline.SqlLine.dispatch(SqlLine.java:813)
>
> at sqlline.SqlLine.begin(SqlLine.java:686)
>
> at sqlline.SqlLine.start(SqlLine.java:398)
>
> at sqlline.SqlLine.main(SqlLine.java:291)
>
> - Will
>