[GitHub] [hbase] busbey commented on a change in pull request #785: HBASE-23239 Reporting on status of backing MOB files from client-facing cells

[GitBox <gi...@apache.org>]

busbey commented on a change in pull request #785: HBASE-23239 Reporting on status of backing MOB files from client-facing cells
URL: https://github.com/apache/hbase/pull/785#discussion_r342941535
 
 

 ##########
 File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/MobRefReporter.java
 ##########
 @@ -0,0 +1,485 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.mob.mapreduce;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.HashSet;
+import java.util.UUID;
+import java.util.Base64;
+
+import com.google.protobuf.ServiceException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.io.HFileLink;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.hadoop.hbase.mapreduce.TableInputFormat;
+import org.apache.hadoop.hbase.mapreduce.TableMapper;
+import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
+import org.apache.hadoop.hbase.mob.MobConstants;
+import org.apache.hadoop.hbase.mob.MobUtils;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.hadoop.hbase.util.HFileArchiveUtil;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.Reducer;
+import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Scans a given table + CF for all mob reference cells to get the list of backing mob files.
+ * For each referenced file we attempt to verify that said file is on the FileSystem in a place
+ * that the MOB system will look when attempting to resolve the actual value.
+ *
+ * The job includes counters that can help provide a rough sketch of the mob data.
+ *
+ * <pre>
+ * Map-Reduce Framework
+ *         Map input records=10000
+ * ...
+ *         Reduce output records=99
+ * ...
+ * CELLS_PER_ROW_DIGITS
+ *         1=10000
+ * MOB
+ *         NUM_CELLS=52364
+ * PROBLEM
+ *         IMPACTED_ROWS=338
+ *         MOB_FILES=2
+ * PROBLEM_ROWS_PER_FILE_DIGITS
+ *         3=2
+ * SIZE_PER_CELL_DIGITS
+ *         5=627
+ *         6=51392
+ *         7=345
+ * SIZE_PER_ROW_DIGITS
+ *         6=6838
+ *         7=3162
+ * </pre>
+ *
+ *   * Map-Reduce Framework:Map input records - the number of rows with mob references
+ *   * Map-Reduce Framework:Reduce output records - the number of unique hfiles referenced
+ *   * MOB:NUM_CELLS - the total number of mob reference cells
+ *   * PROBLEM:IMPACTED_ROWS - the number of rows that reference hfiles with an issue
+ *   * PROBLEM:MOB_FILES - the number of unique hfiles that have an issue
+ *   * CELLS_PER_ROW_DIGITS: - this counter group gives a histogram of the order of magnitude of the
+ *         number of cells in a given row by grouping by the number of digits used in each count.
+ *         This allows us to see more about the distribution of cells than what we can determine
+ *         with just the cell count and the row count. In this particular example we can see that
+ *         all of our rows have somewhere between 1 - 9 cells.
+ *   * PROBLEM_ROWS_PER_FILE_DIGITS: - this counter group gives a histogram of the order of
+ *         magnitude of the number of rows in each of the hfiles with a problem. e.g. in the
+ *         example there are 2 hfiles and they each have the same order of magnitude number of rows,
+ *         namely 3 digits worth i.e. 100-999.
+ *   * SIZE_PER_CELL_DIGITS: - this counter group gives a histogram of the order of magnitude of
+ *         the size of mob values according to our reference cells. e.g. in the example above we
+ *         have between 5 and 7 digits in our cell sizes i.e. 10,000 - 9,999,999 bytes. From this
+ *         histogram we can also see that _most_ cells are 100,000 - 199,000 bytes and the smaller
+ *         and bigger ones are outliers making up less than 2% of mob cells.
+ *   * SIZE_PER_ROW_DIGITS: - this counter group gives a histogram of the order of magnitude of the
+ *         size of
+ *
+ * Generates a report that gives one file status per line, with tabs dividing fields.
+ *
+ * <pre>
+ * RESULT OF LOOKUP	FILE REF	comma seperated, base64 encoded rows when there's a problem
+ * </pre>
+ *
+ * e.g.
+ *
+ * <pre>
+ * MOB DIR	09c576e28a65ed2ead0004d192ffaa382019110184b30a1c7e034573bf8580aef8393402
+ * MISSING FILE    28e252d7f013973174750d483d358fa020191101f73536e7133f4cd3ab1065edf588d509        MmJiMjMyYzBiMTNjNzc0OTY1ZWY4NTU4ZjBmYmQ2MTUtNTIz,MmEzOGE0YTkzMTZjNDllNWE4MzM1MTdjNDVkMzEwNzAtODg=
 
 Review comment:
   base64 can be used in lots of places and the hex output from the hbase shell can basically just be used there.
   
   Would examples for using the base64 output in the hbase shell help? I have some handy.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services