Reading Performance With Multi-Threads

[Bing Li <lb...@gmail.com>]

Dear all,

Pseudo-distributed mode is still used since I am still coding.

When scanning a table, I noticed that a single thread was much faster than
each one in a multi-threads module.

For example, the following method can be done in 2 or 3ms with a single
thread. If 30 threads execute the method together, it takes each thread
about 150ms to execute.

Each of the threads can get a HTableInterface from HTablePool. So I think
the performance should not be so low.

Maybe the pseudo-distributed mode causes the problem?

Thanks so much!

Best regards,
Bing

        public Set<String> GetOutgoingHHNeighborKeys(String hubKey, String
groupKey, int timingScale)
        {
                List<Filter> hhNeighborFilterList = new ArrayList<Filter>();

                SingleColumnValueFilter hubKeyFilter = new
SingleColumnValueFilter(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY,
NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN,
CompareFilter.CompareOp.EQUAL, new SubstringComparator(hubKey));
                hubKeyFilter.setFilterIfMissing(true);
                hhNeighborFilterList.add(hubKeyFilter);

                SingleColumnValueFilter groupKeyFilter = new
SingleColumnValueFilter(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY,
NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN,
CompareFilter.CompareOp.EQUAL, new SubstringComparator(groupKey));
                groupKeyFilter.setFilterIfMissing(true);
                hhNeighborFilterList.add(groupKeyFilter);

                SingleColumnValueFilter timingScaleFilter = new
SingleColumnValueFilter(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY,
NeighborStructure.HUB_HUB_NEIGHBOR_TIMING_SCALE_COLUMN,
CompareFilter.CompareOp.EQUAL, new
BinaryComparator(Bytes.toBytes(timingScale)));
                timingScaleFilter.setFilterIfMissing(true);
                hhNeighborFilterList.add(timingScaleFilter);

                FilterList hhNeighborFilter = new
FilterList(hhNeighborFilterList);
                Scan scan = new Scan();
                scan.setFilter(hhNeighborFilter);
                scan.setCaching(Parameters.CACHING_SIZE);
                scan.setBatch(Parameters.BATCHING_SIZE);

                Set<String> neighborKeySet = Sets.newHashSet();
                String qualifier;
                try
                {
                        ResultScanner scanner =
this.neighborTable.getScanner(scan);
                        for (Result result : scanner)
                        {
                                for (KeyValue kv : result.raw())
                                {
                                        qualifier =
Bytes.toString(kv.getQualifier());
                                        if
(qualifier.equals(NeighborStructure.HUB_HUB_NEIGHBOR_NODE_KEY_STRING_COLUMN))
                                        {

neighborKeySet.add(Bytes.toString(kv.getValue()));
                                        }
                                }
                        }
                        scanner.close();
                }
                catch (IOException e)
                {
                        e.printStackTrace();
                }
                return neighborKeySet;
        }