You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Xiaofeng Wang <ca...@gmail.com> on 2018/03/29 16:15:28 UTC
A strange issue about missing data
Hi, recently I encounter a strange issue about missing data. For instance, column A, B and C is the primary keys in Phoenix and I can get all the relevant records via the condition A=‘aaa'. But some time later(maybe one hour or more), I get the less records than before with the same condition. I don't know why :(
BTW: The Phoenix table is salted with bucket 13
This is my environment:
Phoenix: 4.8.0
HBase: 1.2.0
hbase-site.xml:
<configuration>
<property>
<name>hbase.regionserver.thrift.http</name>
<value>true</value>
</property>
<property>
<name>hbase.thrift.support.proxyuser</name>
<value>true</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://ns1/hbase</value> <hdfs://ns1/hbase%3C/value%3E>
</property>
<property>
<name>hbase.master.port</name>
<value>60000</value>
</property>
<property>
<name>hbase.master.info.port</name>
<value>60010</value>
</property>
<property>
<name>hbase.regionserver.port</name>
<value>60020</value>
</property>
<property>
<name>hbase.regionserver.info.port</name>
<value>60030</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>host1:2181,host2:2181,host3:2181</value>
</property>
<property>
<name>zookeeper.recovery.retry</name>
<value>3</value>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>60000</value>
</property>
<property>
<name>hbase.ipc.server.callqueue.handler.factor</name>
<value>0.2</value>
</property>
<property>
<name>hbase.ipc.server.callqueue.read.ratio</name>
<value>0.3</value>
</property>
<property>
<name>hbase.ipc.server.callqueue.scan.ratio</name>
<value>0.4</value>
</property>
<property>
<name>hbase.client.max.perserver.tasks</name>
<value>7</value>
</property>
<property>
<name>hbase.client.max.perregion.tasks</name>
<value>4</value>
</property>
<property>
<name>hbase.offpeak.start.hour</name>
<value>2</value>
</property>
<property>
<name>hbase.offpeak.end.hour</name>
<value>4</value>
</property>
<property>
<name>hbase.storescanner.parallel.seek.enable</name>
<value>true</value>
</property>
<property>
<name>hbase.storescanner.parallel.seek.threads</name>
<value>10</value>
</property>
<property>
<name>hbase.client.retries.number</name>
<value>5</value>
</property>
<property>
<name>hbase.client.scanner.caching</name>
<value>1000</value>
</property>
<property>
<name>hbase.regionserver.handler.count</name>
<value>100</value>
</property>
<property>
<name>hfile.block.cache.size</name>
<value>0.4</value>
</property>
<property>
<name>hbase.coprocessor.abortonerror</name>
<value>false</value>
</property>
<property>
<name>hbase.regionserver.thrift.framed</name>
<value>false</value>
</property>
<property>
<name>hbase.column.max.version</name>
<value>3</value>
</property>
<property>
<name>hbase.status.published</name>
<value>false</value>
</property>
<property>
<name>hbase.status.multicast.address.port</name>
<value>61000</value>
</property>
<property>
<name>hbase.lease.recovery.dfs.timeout</name>
<value>2300000</value>
</property>
<property>
<name>dfs.client.sockettimeout</name>
<value>1000000</value>
</property>
<property>
<name>hbase.master.distributed.log.replay</name>
<value>true</value>
</property>
<property>
<name>hbase.rpc.engine</name>
<value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
</property>
<property>
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.hadoop.hbase.security.access.AccessController</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
</property>
<property>
<name>hbase.superuser</name>
<value>hbase</value>
</property>
<property>
<name>hbase.security.authorization</name>
<value>false</value>
</property>
<property>
<name>hbase.client.write.buffer</name>
<value>5242880</value>
</property>
<property>
<name>hbase.hregion.max.filesize</name>
<value>10737418240</value>
</property>
<property>
<name>hbase.rpc.timeout</name>
<value>1000000</value>
</property>
<property>
<name>hbase.regionserver.wal.codec</name>
<value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
</property>
<property>
<name>hbase.client.keyvalue.maxsize</name>
<value>20971520</value>
</property>
<property>
<name>hbase.hregion.memstore.flush.size</name>
<value>536870912</value>
</property>
<property>
<name>hbase.hregion.memstore.block.multiplier</name>
<value>8</value>
</property>
<property>
<name>hbase.hstore.blockingStoreFiles</name>
<value>300</value>
</property>
<property>
<name>hbase.hstore.compactionThreshold</name>
<value>10</value>
</property>
<property>
<name>hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily</name>
<value>320</value>
</property>
</configuration>
Thank you all!
Re: A strange issue about missing data
Posted by Xiaofeng Wang <ca...@gmail.com>.
Thanks Sergey. I forgot to mention that the data is loaded via MapReduce. If I load data with HFile, will Phoenix generate the stats automatically? This is what I didn’t understand. I find this on official page "Statistics are also automatically collected during major compactions and region splits so manually running this command may not be necessary."
> On 30 Mar 2018, at 12:44 AM, Sergey Soldatov <se...@gmail.com> wrote:
>
> Usually such kind of problems may happen when something wrong with the statistic. You may try to clean SYSTEM.STATS, restart the client and check whether it fixes the problem. If not, you may try to turn on DEBUG log level on the client and check whether generated scans are covering all regions (this task is easier to perform with clean stats).
>
> Thanks,
> Sergey
>
>
> On Thu, Mar 29, 2018 at 9:15 AM, Xiaofeng Wang <capt7w@gmail.com <ma...@gmail.com>> wrote:
> Hi, recently I encounter a strange issue about missing data. For instance, column A, B and C is the primary keys in Phoenix and I can get all the relevant records via the condition A=‘aaa'. But some time later(maybe one hour or more), I get the less records than before with the same condition. I don't know why :(
>
> BTW: The Phoenix table is salted with bucket 13
>
> This is my environment:
>
> Phoenix: 4.8.0
> HBase: 1.2.0
>
> hbase-site.xml:
>
> <configuration>
> <property>
> <name>hbase.regionserver.thrift.http</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.thrift.support.proxyuser</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://ns1/hbase</value> <>
> </property>
> <property>
> <name>hbase.master.port</name>
> <value>60000</value>
> </property>
> <property>
> <name>hbase.master.info <http://hbase.master.info/>.port</name>
> <value>60010</value>
> </property>
> <property>
> <name>hbase.regionserver.port</name>
> <value>60020</value>
> </property>
> <property>
> <name>hbase.regionserver.info.port</name>
> <value>60030</value>
> </property>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>host1:2181,host2:2181,host3:2181</value>
> </property>
> <property>
> <name>zookeeper.recovery.retry</name>
> <value>3</value>
> </property>
> <property>
> <name>zookeeper.session.timeout</name>
> <value>60000</value>
> </property>
> <property>
> <name>hbase.ipc.server.callqueue.handler.factor</name>
> <value>0.2</value>
> </property>
> <property>
> <name>hbase.ipc.server.callqueue.read.ratio</name>
> <value>0.3</value>
> </property>
> <property>
> <name>hbase.ipc.server.callqueue.scan.ratio</name>
> <value>0.4</value>
> </property>
> <property>
> <name>hbase.client.max.perserver.tasks</name>
> <value>7</value>
> </property>
> <property>
> <name>hbase.client.max.perregion.tasks</name>
> <value>4</value>
> </property>
> <property>
> <name>hbase.offpeak.start.hour</name>
> <value>2</value>
> </property>
> <property>
> <name>hbase.offpeak.end.hour</name>
> <value>4</value>
> </property>
> <property>
> <name>hbase.storescanner.parallel.seek.enable</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.storescanner.parallel.seek.threads</name>
> <value>10</value>
> </property>
> <property>
> <name>hbase.client.retries.number</name>
> <value>5</value>
> </property>
> <property>
> <name>hbase.client.scanner.caching</name>
> <value>1000</value>
> </property>
> <property>
> <name>hbase.regionserver.handler.count</name>
> <value>100</value>
> </property>
> <property>
> <name>hfile.block.cache.size</name>
> <value>0.4</value>
> </property>
> <property>
> <name>hbase.coprocessor.abortonerror</name>
> <value>false</value>
> </property>
> <property>
> <name>hbase.regionserver.thrift.framed</name>
> <value>false</value>
> </property>
> <property>
> <name>hbase.column.max.version</name>
> <value>3</value>
> </property>
> <property>
> <name>hbase.status.published</name>
> <value>false</value>
> </property>
> <property>
> <name>hbase.status.multicast.address.port</name>
> <value>61000</value>
> </property>
> <property>
> <name>hbase.lease.recovery.dfs.timeout</name>
> <value>2300000</value>
> </property>
> <property>
> <name>dfs.client.sockettimeout</name>
> <value>1000000</value>
> </property>
> <property>
> <name>hbase.master.distributed.log.replay</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.rpc.engine</name>
> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
> </property>
> <property>
> <name>hbase.coprocessor.master.classes</name>
> <value>org.apache.hadoop.hbase.security.access.AccessController</value>
> </property>
> <property>
> <name>hbase.coprocessor.region.classes</name>
> <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
> </property>
> <property>
> <name>hbase.superuser</name>
> <value>hbase</value>
> </property>
> <property>
> <name>hbase.security.authorization</name>
> <value>false</value>
> </property>
> <property>
> <name>hbase.client.write.buffer</name>
> <value>5242880</value>
> </property>
> <property>
> <name>hbase.hregion.max.filesize</name>
> <value>10737418240</value>
> </property>
> <property>
> <name>hbase.rpc.timeout</name>
> <value>1000000</value>
> </property>
> <property>
> <name>hbase.regionserver.wal.codec</name>
> <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
> </property>
> <property>
> <name>hbase.client.keyvalue.maxsize</name>
> <value>20971520</value>
> </property>
> <property>
> <name>hbase.hregion.memstore.flush.size</name>
> <value>536870912</value>
> </property>
> <property>
> <name>hbase.hregion.memstore.block.multiplier</name>
> <value>8</value>
> </property>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>300</value>
> </property>
> <property>
> <name>hbase.hstore.compactionThreshold</name>
> <value>10</value>
> </property>
> <property>
> <name>hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily</name>
> <value>320</value>
> </property>
> </configuration>
>
> Thank you all!
>
Re: A strange issue about missing data
Posted by Sergey Soldatov <se...@gmail.com>.
Usually such kind of problems may happen when something wrong with the
statistic. You may try to clean SYSTEM.STATS, restart the client and check
whether it fixes the problem. If not, you may try to turn on DEBUG log
level on the client and check whether generated scans are covering all
regions (this task is easier to perform with clean stats).
Thanks,
Sergey
On Thu, Mar 29, 2018 at 9:15 AM, Xiaofeng Wang <ca...@gmail.com> wrote:
> Hi, recently I encounter a strange issue about missing data. For instance,
> column A, B and C is the primary keys in Phoenix and I can get all the
> relevant records via the condition A=‘aaa'. But some time later(maybe one
> hour or more), I get the less records than before with the same condition.
> I don't know why :(
>
> BTW: The Phoenix table is salted with bucket 13
>
> This is my environment:
>
> Phoenix: 4.8.0
> HBase: 1.2.0
>
> hbase-site.xml:
>
> <configuration>
> <property>
> <name>hbase.regionserver.thrift.http</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.thrift.support.proxyuser</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://ns1/hbase</value>
> </property>
> <property>
> <name>hbase.master.port</name>
> <value>60000</value>
> </property>
> <property>
> <name>hbase.master.info.port</name>
> <value>60010</value>
> </property>
> <property>
> <name>hbase.regionserver.port</name>
> <value>60020</value>
> </property>
> <property>
> <name>hbase.regionserver.info.port</name>
> <value>60030</value>
> </property>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>host1:2181,host2:2181,host3:2181</value>
> </property>
> <property>
> <name>zookeeper.recovery.retry</name>
> <value>3</value>
> </property>
> <property>
> <name>zookeeper.session.timeout</name>
> <value>60000</value>
> </property>
> <property>
> <name>hbase.ipc.server.callqueue.handler.factor</name>
> <value>0.2</value>
> </property>
> <property>
> <name>hbase.ipc.server.callqueue.read.ratio</name>
> <value>0.3</value>
> </property>
> <property>
> <name>hbase.ipc.server.callqueue.scan.ratio</name>
> <value>0.4</value>
> </property>
> <property>
> <name>hbase.client.max.perserver.tasks</name>
> <value>7</value>
> </property>
> <property>
> <name>hbase.client.max.perregion.tasks</name>
> <value>4</value>
> </property>
> <property>
> <name>hbase.offpeak.start.hour</name>
> <value>2</value>
> </property>
> <property>
> <name>hbase.offpeak.end.hour</name>
> <value>4</value>
> </property>
> <property>
> <name>hbase.storescanner.parallel.seek.enable</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.storescanner.parallel.seek.threads</name>
> <value>10</value>
> </property>
> <property>
> <name>hbase.client.retries.number</name>
> <value>5</value>
> </property>
> <property>
> <name>hbase.client.scanner.caching</name>
> <value>1000</value>
> </property>
> <property>
> <name>hbase.regionserver.handler.count</name>
> <value>100</value>
> </property>
> <property>
> <name>hfile.block.cache.size</name>
> <value>0.4</value>
> </property>
> <property>
> <name>hbase.coprocessor.abortonerror</name>
> <value>false</value>
> </property>
> <property>
> <name>hbase.regionserver.thrift.framed</name>
> <value>false</value>
> </property>
> <property>
> <name>hbase.column.max.version</name>
> <value>3</value>
> </property>
> <property>
> <name>hbase.status.published</name>
> <value>false</value>
> </property>
> <property>
> <name>hbase.status.multicast.address.port</name>
> <value>61000</value>
> </property>
> <property>
> <name>hbase.lease.recovery.dfs.timeout</name>
> <value>2300000</value>
> </property>
> <property>
> <name>dfs.client.sockettimeout</name>
> <value>1000000</value>
> </property>
> <property>
> <name>hbase.master.distributed.log.replay</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.rpc.engine</name>
> <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
> </property>
> <property>
> <name>hbase.coprocessor.master.classes</name>
> <value>org.apache.hadoop.hbase.security.access.AccessController</value>
> </property>
> <property>
> <name>hbase.coprocessor.region.classes</name>
> <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.
> hadoop.hbase.security.access.AccessController</value>
> </property>
> <property>
> <name>hbase.superuser</name>
> <value>hbase</value>
> </property>
> <property>
> <name>hbase.security.authorization</name>
> <value>false</value>
> </property>
> <property>
> <name>hbase.client.write.buffer</name>
> <value>5242880</value>
> </property>
> <property>
> <name>hbase.hregion.max.filesize</name>
> <value>10737418240</value>
> </property>
> <property>
> <name>hbase.rpc.timeout</name>
> <value>1000000</value>
> </property>
> <property>
> <name>hbase.regionserver.wal.codec</name>
> <value>org.apache.hadoop.hbase.regionserver.wal.
> IndexedWALEditCodec</value>
> </property>
> <property>
> <name>hbase.client.keyvalue.maxsize</name>
> <value>20971520</value>
> </property>
> <property>
> <name>hbase.hregion.memstore.flush.size</name>
> <value>536870912</value>
> </property>
> <property>
> <name>hbase.hregion.memstore.block.multiplier</name>
> <value>8</value>
> </property>
> <property>
> <name>hbase.hstore.blockingStoreFiles</name>
> <value>300</value>
> </property>
> <property>
> <name>hbase.hstore.compactionThreshold</name>
> <value>10</value>
> </property>
> <property>
> <name>hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily</name>
> <value>320</value>
> </property>
> </configuration>
>
> Thank you all!
>