You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by ns...@apache.org on 2011/10/11 19:44:18 UTC
svn commit: r1181958 - in /hbase/branches/0.89/src/main:
java/org/apache/hadoop/hbase/client/ java/org/apache/hadoop/hbase/ipc/
java/org/apache/hadoop/hbase/master/
java/org/apache/hadoop/hbase/regionserver/ ruby/hbase/
Author: nspiegelberg
Date: Tue Oct 11 17:44:18 2011
New Revision: 1181958
URL: http://svn.apache.org/viewvc?rev=1181958&view=rev
Log:
Batch CF Updates on table alters
Summary:
HBaseAdmin and HMasterInterface provide no way to batch alterations to a table.
This results in RPC inefficiency, wasteful extra writes to META and, with the
introduction of online schema change, a much worse problem that all regions have
to be restarted for each separate change in a single table alter.
This patch includes a new ColumnOperation, MultiColumnOperation, that performs
all of the changes requested for each region at the same time, and performs a
single write to META for each region. It also includes necessary reorganization
of the existing ColumnOperations to enable this functionality.
It also exposes new methods to both HBaseAdmin clients and HMasterInterface RPC
clients so that they can make batch calls in order to make use of this batched
update feature.
A few design decisions that could be controversial:
1. Four method prototypes are exposed for each type of operation (add, modify,
delete) in HBaseAdmin, plus two for the batched operation. This is in order to
allow easy per-operation batching and in order to not require the user to
perform String-to-bytes conversions. It also follows the current conventions in
HBaseAdmin, which exposes String equivalents for each non-batched operation.
However, it is a possibility that the confusion of having so many options for
each operation is not worth the added convenience. I'm up for suggestions here.
2. Used three lists (one each for add, delete, modify) rather than some kind of
new "MultiAlter" type. This is also for simplicity to the user, but the tradeoff
is not being able to, say, perform String-to-bytes conversions automatically, or
to reduce the number of required method prototypes. Once again, certainly
interested in suggestions!
Test Plan:
Ran all unit tests. No failures.
Basic sanity tests on dev server:
Prior to Patch, for N changes in a single alter, we see N updates, complete
with N region restarts:
hbase(main):003:0> alter 'peeps', {NAME => 'demographics'}, {NAME => 'name',
VERSIONS => '1'}, {NAME => 'location', METHOD => 'delete'}
Updating all regions with the new schema...
0/1 regions updated.
0/1 regions updated.
0/0 regions updated.
Done.
Updating all regions with the new schema...
0/1 regions updated.
0/1 regions updated.
0/0 regions updated.
Done.
Updating all regions with the new schema...
0/1 regions updated.
0/1 regions updated.
0/0 regions updated.
Done.
0 row(s) in 9.2600 seconds
After patch, for N alterations, we have one update:
hbase(main):009:0> alter 'peeps', {NAME => 'demographics', METHOD => 'delete'},
{NAME => 'name', VERSIONS => '2'}, {NAME => 'location', VERSIONS => '1'}
Updating all regions with the new schema...
0/1 regions updated.
0/1 regions updated.
0/0 regions updated.
Done.
0 row(s) in 3.0920 seconds
Reviewers: nspiegelberg, kannan, kranganathan
Reviewed By: kannan
CC: hbase@lists, kannan
Differential Revision: 314294
Task ID: 678576
Added:
hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/MultiColumnOperation.java
Modified:
hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/AddColumn.java
hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/ColumnOperation.java
hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/DeleteColumn.java
hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/ModifyColumn.java
hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/TableOperation.java
hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
hbase/branches/0.89/src/main/ruby/hbase/admin.rb
Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java?rev=1181958&r1=1181957&r2=1181958&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java (original)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java Tue Oct 11 17:44:18 2011
@@ -23,7 +23,9 @@ import java.io.InterruptedIOException;
import java.io.IOException;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.Arrays;
+import java.util.ArrayList;
import java.util.Iterator;
+import java.util.List;
import java.util.Map;
import java.util.NavigableMap;
import java.util.Set;
@@ -576,6 +578,69 @@ public class HBaseAdmin {
}
/**
+ * Batch alter a table. Only takes regions offline once and performs a single
+ * update to .META.
+ * Asynchronous operation.
+ *
+ * @param tableName name of the table to add column to
+ * @param columnAdditions column descriptors to add to the table
+ * @param columnModifications pairs of column names with new descriptors
+ * @param columnDeletions column names to delete from the table
+ * @throws IOException if a remote or network exception occurs
+ */
+ public void alterTable(final String tableName,
+ List<HColumnDescriptor> columnAdditions,
+ List<Pair<String, HColumnDescriptor>> columnModifications,
+ List<String> columnDeletions) throws IOException {
+ // convert all of the strings to bytes and pass to the bytes method
+ List<Pair<byte [], HColumnDescriptor>> modificationsBytes =
+ new ArrayList<Pair<byte [], HColumnDescriptor>>(
+ columnModifications.size());
+ List<byte []> deletionsBytes =
+ new ArrayList<byte []>(columnDeletions.size());
+
+ for(Pair<String, HColumnDescriptor> c : columnModifications) {
+ modificationsBytes.add(new Pair<byte [], HColumnDescriptor>(
+ Bytes.toBytes(c.getFirst()), c.getSecond()));
+ }
+ for(String c : columnDeletions) {
+ deletionsBytes.add(Bytes.toBytes(c));
+ }
+
+ alterTable(Bytes.toBytes(tableName), columnAdditions, modificationsBytes,
+ deletionsBytes);
+ }
+
+ /**
+ * Batch alter a table. Only takes regions offline once and performs a single
+ * update to .META.
+ * Any of the three lists can be null, in which case those types of
+ * alterations will be ignored.
+ * Asynchronous operation.
+ *
+ * @param tableName name of the table to add column to
+ * @param columnAdditions column descriptors to add to the table
+ * @param columnModifications pairs of column names with new descriptors
+ * @param columnDeletions column names to delete from the table
+ * @throws IOException if a remote or network exception occurs
+ */
+ public void alterTable(final byte [] tableName,
+ List<HColumnDescriptor> columnAdditions,
+ List<Pair<byte[], HColumnDescriptor>> columnModifications,
+ List<byte[]> columnDeletions) throws IOException {
+ if (this.master == null) {
+ throw new MasterNotRunningException("master has been shut down");
+ }
+ HTableDescriptor.isLegalTableName(tableName);
+ try {
+ this.master.alterTable(tableName, columnAdditions, columnModifications,
+ columnDeletions);
+ } catch (RemoteException e) {
+ throw RemoteExceptionHandler.decodeRemoteException(e);
+ }
+ }
+
+ /**
* Get the status of alter command - indicates how many regions have received
* the updated schema Asynchronous operation.
*
@@ -607,7 +672,7 @@ public class HBaseAdmin {
*/
public void addColumn(final String tableName, HColumnDescriptor column)
throws IOException {
- addColumn(Bytes.toBytes(tableName), column);
+ alterTable(Bytes.toBytes(tableName), Arrays.asList(column), null, null);
}
/**
@@ -620,15 +685,7 @@ public class HBaseAdmin {
*/
public void addColumn(final byte [] tableName, HColumnDescriptor column)
throws IOException {
- if (this.master == null) {
- throw new MasterNotRunningException("master has been shut down");
- }
- HTableDescriptor.isLegalTableName(tableName);
- try {
- this.master.addColumn(tableName, column);
- } catch (RemoteException e) {
- throw RemoteExceptionHandler.decodeRemoteException(e);
- }
+ alterTable(tableName, Arrays.asList(column), null, null);
}
/**
@@ -641,7 +698,8 @@ public class HBaseAdmin {
*/
public void deleteColumn(final String tableName, final String columnName)
throws IOException {
- deleteColumn(Bytes.toBytes(tableName), Bytes.toBytes(columnName));
+ alterTable(Bytes.toBytes(tableName), null, null,
+ Arrays.asList(Bytes.toBytes(columnName)));
}
/**
@@ -652,17 +710,10 @@ public class HBaseAdmin {
* @param columnName name of column to be deleted
* @throws IOException if a remote or network exception occurs
*/
- public void deleteColumn(final byte [] tableName, final byte [] columnName)
+ public void deleteColumn(final byte [] tableName,
+ final byte [] columnName)
throws IOException {
- if (this.master == null) {
- throw new MasterNotRunningException("master has been shut down");
- }
- HTableDescriptor.isLegalTableName(tableName);
- try {
- this.master.deleteColumn(tableName, columnName);
- } catch (RemoteException e) {
- throw RemoteExceptionHandler.decodeRemoteException(e);
- }
+ alterTable(tableName, null, null, Arrays.asList(columnName));
}
/**
@@ -677,8 +728,9 @@ public class HBaseAdmin {
public void modifyColumn(final String tableName, final String columnName,
HColumnDescriptor descriptor)
throws IOException {
- modifyColumn(Bytes.toBytes(tableName), Bytes.toBytes(columnName),
- descriptor);
+ alterTable(Bytes.toBytes(tableName), null, Arrays.asList(
+ new Pair<byte [], HColumnDescriptor>(Bytes.toBytes(columnName),
+ descriptor)), null);
}
/**
@@ -693,15 +745,8 @@ public class HBaseAdmin {
public void modifyColumn(final byte [] tableName, final byte [] columnName,
HColumnDescriptor descriptor)
throws IOException {
- if (this.master == null) {
- throw new MasterNotRunningException("master has been shut down");
- }
- HTableDescriptor.isLegalTableName(tableName);
- try {
- this.master.modifyColumn(tableName, columnName, descriptor);
- } catch (RemoteException e) {
- throw RemoteExceptionHandler.decodeRemoteException(e);
- }
+ alterTable(tableName, null, Arrays.asList(
+ new Pair<byte [], HColumnDescriptor>(columnName, descriptor)), null);
}
/**
Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java?rev=1181958&r1=1181957&r2=1181958&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java (original)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java Tue Oct 11 17:44:18 2011
@@ -19,6 +19,8 @@
*/
package org.apache.hadoop.hbase.ipc;
+import java.util.List;
+
import org.apache.hadoop.hbase.ClusterStatus;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HConstants;
@@ -64,6 +66,21 @@ public interface HMasterInterface extend
public void deleteTable(final byte [] tableName) throws IOException;
/**
+ * Batch adds, modifies, and deletes columns from the specified table.
+ * Any of the lists may be null, in which case those types of alterations
+ * will not occur.
+ * @param tableName table to modify
+ * @param columnAdditions column descriptors to add to the table
+ * @param columnModifications pairs of column names with new descriptors
+ * @param columnDeletions column names to delete from the table
+ * @throws IOException e
+ */
+ public void alterTable(final byte [] tableName,
+ List<HColumnDescriptor> columnAdditions,
+ List<Pair<byte[], HColumnDescriptor>> columnModifications,
+ List<byte[]> columnDeletions) throws IOException;
+
+ /**
* Adds a column to the specified table
* @param tableName table to modify
* @param column column descriptor
@@ -83,7 +100,6 @@ public interface HMasterInterface extend
HColumnDescriptor descriptor)
throws IOException;
-
/**
* Deletes a column from the specified table. Table must be disabled.
* @param tableName table to alter
Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/AddColumn.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/AddColumn.java?rev=1181958&r1=1181957&r2=1181958&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/AddColumn.java (original)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/AddColumn.java Tue Oct 11 17:44:18 2011
@@ -19,21 +19,14 @@
*/
package org.apache.hadoop.hbase.master;
-import org.apache.hadoop.hbase.HColumnDescriptor;
-import org.apache.hadoop.hbase.HRegionInfo;
-import org.apache.hadoop.hbase.ipc.HRegionInterface;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
-
import java.io.IOException;
-import java.util.HashSet;
-import java.util.Set;
+
+import org.apache.hadoop.hbase.HColumnDescriptor;
+import org.apache.hadoop.hbase.HTableDescriptor;
/** Instantiated to add a column family to a table */
class AddColumn extends ColumnOperation {
private final HColumnDescriptor newColumn;
- private static final Log LOG = LogFactory.getLog(AddColumn.class);
AddColumn(final HMaster master, final byte [] tableName,
final HColumnDescriptor newColumn)
@@ -43,33 +36,8 @@ class AddColumn extends ColumnOperation
}
@Override
- protected void postProcessMeta(MetaRegion m, HRegionInterface server)
+ protected void updateTableDescriptor(HTableDescriptor desc)
throws IOException {
- Set<HRegionInfo> regionsToReopen = new HashSet<HRegionInfo>();
- for (HRegionInfo i : regionsToProcess) {
- // All we need to do to add a column is add it to the table descriptor.
- // When the region is brought on-line, it will find the column missing
- // and create it.
- i.getTableDesc().addFamily(newColumn);
- updateRegionInfo(server, m.getRegionName(), i);
- // Ignore regions that are split or disabled,
- // as we do not want to reopen them
- if (!(i.isSplit() || i.isOffline())) {
- regionsToReopen.add(i);
- }
- }
- if (regionsToReopen.size() > 0) {
- this.master.getRegionManager().getThrottledReopener(Bytes.toString(tableName)).
- addRegionsToReopen(regionsToReopen);
- }
- }
-
- @Override
- protected void processScanItem(String serverName, final HRegionInfo info)
- throws IOException {
- if (isEnabled(info)) {
- LOG.debug("Performing online schema change (region not disabled): "
- + info.getRegionNameAsString());
- }
+ desc.addFamily(newColumn);
}
-}
\ No newline at end of file
+}
Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/ColumnOperation.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/ColumnOperation.java?rev=1181958&r1=1181957&r2=1181958&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/ColumnOperation.java (original)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/ColumnOperation.java Tue Oct 11 17:44:18 2011
@@ -19,13 +19,24 @@
*/
package org.apache.hadoop.hbase.master;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HColumnDescriptor;
+import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.TableNotDisabledException;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.ipc.HRegionInterface;
+import org.apache.hadoop.hbase.regionserver.Store;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.hbase.util.Writables;
import java.io.IOException;
@@ -40,9 +51,10 @@ abstract class ColumnOperation extends T
@Override
protected void processScanItem(String serverName, final HRegionInfo info)
- throws IOException {
+ throws IOException {
if (isEnabled(info)) {
- throw new TableNotDisabledException(tableName);
+ LOG.debug("Performing online schema change (region not disabled): "
+ + info.getRegionNameAsString());
}
}
@@ -55,4 +67,51 @@ abstract class ColumnOperation extends T
LOG.debug("updated columns in row: " + i.getRegionNameAsString());
}
}
+
+ /**
+ * Simply updates the given table descriptor with the relevant changes
+ * for the given column operation
+ * @param desc The descriptor that will be updated.
+ */
+ protected abstract void updateTableDescriptor(HTableDescriptor desc)
+ throws IOException;
+
+ /**
+ * Is run after META has been updated. Defaults to doing nothing, but
+ * some operations make use of this for housekeeping operations.
+ * @param hri Info for the region that has just been updated.
+ */
+ protected void postProcess(HRegionInfo hri) throws IOException {
+ // do nothing by default
+ }
+
+ /**
+ * Contains all of the logic for updating meta for each type of possible
+ * schema change. By implementing the logic at this level, we are able to
+ * easily prevent excess writes to META
+ * @param m the region
+ */
+ @Override
+ protected void postProcessMeta(MetaRegion m, HRegionInterface server)
+ throws IOException {
+ Set<HRegionInfo> regionsToReopen = new HashSet<HRegionInfo>();
+ for (HRegionInfo i: regionsToProcess) {
+ // All we need to do to change the schema is modify the table descriptor.
+ // When the region is brought on-line, it will find the changes and
+ // update itself accordingly.
+ updateTableDescriptor(i.getTableDesc());
+ updateRegionInfo(server, m.getRegionName(), i);
+ // TODO: queue this to occur after reopening region
+ postProcess(i);
+ // Ignore regions that are split or disabled,
+ // as we do not want to reopen them
+ if (!(i.isSplit() || i.isOffline())) {
+ regionsToReopen.add(i);
+ }
+ }
+ if (regionsToReopen.size() > 0) {
+ this.master.getRegionManager().getThrottledReopener(
+ Bytes.toString(tableName)).addRegionsToReopen(regionsToReopen);
+ }
+ }
}
Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/DeleteColumn.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/DeleteColumn.java?rev=1181958&r1=1181957&r2=1181958&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/DeleteColumn.java (original)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/DeleteColumn.java Tue Oct 11 17:44:18 2011
@@ -19,60 +19,44 @@
*/
package org.apache.hadoop.hbase.master;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
+import java.io.IOException;
+
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HRegionInfo;
-import org.apache.hadoop.hbase.ipc.HRegionInterface;
+import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.regionserver.Store;
import org.apache.hadoop.hbase.util.Bytes;
-import java.io.IOException;
-import java.util.HashSet;
-import java.util.Set;
-
/** Instantiated to remove a column family from a table */
class DeleteColumn extends ColumnOperation {
- private final byte [] columnName;
- private static final Log LOG = LogFactory.getLog(DeleteColumn.class);
+ private final byte [] columnDeletion;
DeleteColumn(final HMaster master, final byte [] tableName,
- final byte [] columnName)
+ final byte [] columnDeletion)
throws IOException {
super(master, tableName);
- this.columnName = columnName;
+ this.columnDeletion = columnDeletion;
}
@Override
- protected void postProcessMeta(MetaRegion m, HRegionInterface server)
- throws IOException {
- Set<HRegionInfo> regionsToReopen = new HashSet<HRegionInfo>();
- for (HRegionInfo i: regionsToProcess) {
- i.getTableDesc().removeFamily(columnName);
- updateRegionInfo(server, m.getRegionName(), i);
- // Delete the directories used by the column
- Path tabledir =
- new Path(this.master.getRootDir(), i.getTableDesc().getNameAsString());
- this.master.getFileSystem().
- delete(Store.getStoreHomedir(tabledir, i.getEncodedName(),
- this.columnName), true);
- // Ignore regions that are split or disabled,
- // as we do not want to reopen them
- if (!(i.isSplit() || i.isOffline())) {
- regionsToReopen.add(i);
- }
- }
- if (regionsToReopen.size() > 0) {
- this.master.getRegionManager().getThrottledReopener(Bytes.toString(tableName)).
- addRegionsToReopen(regionsToReopen);
- }
+ protected void postProcess(HRegionInfo hri) throws IOException {
+ // Delete the directories used by the column
+ Path tabledir = new Path(
+ this.master.getRootDir(), hri.getTableDesc().getNameAsString());
+ this.master.getFileSystem().
+ delete(Store.getStoreHomedir(tabledir, hri.getEncodedName(),
+ columnDeletion), true);
}
+
@Override
- protected void processScanItem(String serverName, final HRegionInfo info)
- throws IOException {
- if (isEnabled(info)) {
- LOG.debug("Performing online schema change (region not disabled): "
- + info.getRegionNameAsString());
+ protected void updateTableDescriptor(HTableDescriptor desc)
+ throws IOException {
+ if (!desc.hasFamily(columnDeletion)) {
+ // we have an error.
+ throw new InvalidColumnNameException("Column family '" +
+ Bytes.toStringBinary(columnDeletion) +
+ "' doesn't exist, so cannot be deleted.");
}
+ desc.removeFamily(columnDeletion);
}
-}
\ No newline at end of file
+}
Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/HMaster.java?rev=1181958&r1=1181957&r2=1181958&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/HMaster.java (original)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Tue Oct 11 17:44:18 2011
@@ -25,6 +25,7 @@ import java.lang.management.ManagementFa
import java.lang.management.RuntimeMXBean;
import java.lang.reflect.Constructor;
import java.net.UnknownHostException;
+import java.util.Arrays;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
@@ -970,12 +971,15 @@ public class HMaster extends Thread impl
}
@Override
- public void addColumn(byte [] tableName, HColumnDescriptor column)
- throws IOException {
+ public void alterTable(final byte [] tableName,
+ List<HColumnDescriptor> columnAdditions,
+ List<Pair<byte[], HColumnDescriptor>> columnModifications,
+ List<byte[]> columnDeletions) throws IOException {
ThrottledRegionReopener reopener = this.regionManager.
createThrottledReopener(Bytes.toString(tableName));
- // Regions are added to the reopener in AddColumn
- new AddColumn(this, tableName, column).process();
+ // Regions are added to the reopener in MultiColumnOperation
+ new MultiColumnOperation(this, tableName, columnAdditions,
+ columnModifications, columnDeletions).process();
reopener.reOpenRegionsThrottle();
}
@@ -992,24 +996,24 @@ public class HMaster extends Thread impl
}
@Override
+ public void addColumn(byte [] tableName, HColumnDescriptor column)
+ throws IOException {
+ alterTable(tableName, Arrays.asList(column), null, null);
+ }
+
+ @Override
public void modifyColumn(byte [] tableName, byte [] columnName,
HColumnDescriptor descriptor)
throws IOException {
- ThrottledRegionReopener reopener = this.regionManager.
- createThrottledReopener(Bytes.toString(tableName));
- // Regions are added to the reopener in ModifyColumn
- new ModifyColumn(this, tableName, columnName, descriptor).process();
- reopener.reOpenRegionsThrottle();
+ alterTable(tableName, null, Arrays.asList(
+ new Pair<byte [], HColumnDescriptor>(columnName, descriptor)), null);
}
@Override
public void deleteColumn(final byte [] tableName, final byte [] c)
throws IOException {
- ThrottledRegionReopener reopener = this.regionManager.
- createThrottledReopener(Bytes.toString(tableName));
- // Regions are added to the reopener in DeleteColumn
- new DeleteColumn(this, tableName, KeyValue.parseColumn(c)[0]).process();
- reopener.reOpenRegionsThrottle();
+ alterTable(tableName, null, null,
+ Arrays.asList(KeyValue.parseColumn(c)[0]));
}
@Override
Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/ModifyColumn.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/ModifyColumn.java?rev=1181958&r1=1181957&r2=1181958&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/ModifyColumn.java (original)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/ModifyColumn.java Tue Oct 11 17:44:18 2011
@@ -19,62 +19,34 @@
*/
package org.apache.hadoop.hbase.master;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
+import java.io.IOException;
+
import org.apache.hadoop.hbase.HColumnDescriptor;
-import org.apache.hadoop.hbase.HRegionInfo;
-import org.apache.hadoop.hbase.ipc.HRegionInterface;
+import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.util.Bytes;
-import java.io.IOException;
-import java.util.Set;
-import java.util.HashSet;
-
/** Instantiated to modify an existing column family on a table */
class ModifyColumn extends ColumnOperation {
- private final HColumnDescriptor descriptor;
- private final byte [] columnName;
- private static final Log LOG = LogFactory.getLog(ModifyColumn.class);
+ private final HColumnDescriptor newColumn;
+ private final byte [] oldColumnName;
ModifyColumn(final HMaster master, final byte [] tableName,
- final byte [] columnName, HColumnDescriptor descriptor)
+ final byte [] oldColumnName, HColumnDescriptor newColumn)
throws IOException {
super(master, tableName);
- this.descriptor = descriptor;
- this.columnName = columnName;
+ this.newColumn = newColumn;
+ this.oldColumnName = oldColumnName;
}
@Override
- protected void postProcessMeta(MetaRegion m, HRegionInterface server)
+ protected void updateTableDescriptor(HTableDescriptor desc)
throws IOException {
- Set<HRegionInfo> regionsToReopen = new HashSet<HRegionInfo>();
- for (HRegionInfo i: regionsToProcess) {
- if (i.getTableDesc().hasFamily(columnName)) {
- i.getTableDesc().addFamily(descriptor);
- updateRegionInfo(server, m.getRegionName(), i);
- // Ignore regions that are split or disabled,
- // as we do not want to reopen them
- if (!(i.isSplit() || i.isOffline())) {
- regionsToReopen.add(i);
- }
- } else { // otherwise, we have an error.
- throw new InvalidColumnNameException("Column family '" +
- Bytes.toString(columnName) +
- "' doesn't exist, so cannot be modified.");
- }
- }
- if (regionsToReopen.size() > 0) {
- this.master.getRegionManager().getThrottledReopener(Bytes.toString(tableName)).
- addRegionsToReopen(regionsToReopen);
- }
- }
-
- @Override
- protected void processScanItem(String serverName, final HRegionInfo info)
- throws IOException {
- if (isEnabled(info)) {
- LOG.debug("Performing online schema change (region not disabled): "
- + info.getRegionNameAsString());
+ if (!desc.hasFamily(oldColumnName)) {
+ // we have an error.
+ throw new InvalidColumnNameException("Column family '" +
+ Bytes.toStringBinary(oldColumnName) +
+ "' doesn't exist, so cannot be modified.");
}
+ desc.addFamily(newColumn);
}
}
Added: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/MultiColumnOperation.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/MultiColumnOperation.java?rev=1181958&view=auto
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/MultiColumnOperation.java (added)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/MultiColumnOperation.java Tue Oct 11 17:44:18 2011
@@ -0,0 +1,78 @@
+/**
+ * Copyright 2010 The Apache Software Foundation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.hbase.HColumnDescriptor;
+import org.apache.hadoop.hbase.HTableDescriptor;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.util.Pair;
+
+import java.io.IOException;
+
+/** Instantiated to process a batch of table alterations */
+class MultiColumnOperation extends ColumnOperation {
+ private final List<ColumnOperation> operations;
+
+ MultiColumnOperation(final HMaster master, final byte [] tableName,
+ final List<HColumnDescriptor> columnAdditions,
+ final List<Pair<byte [], HColumnDescriptor>> columnModifications,
+ final List<byte []> columnDeletions) throws IOException {
+ super(master, tableName);
+ // convert the three separate lists to an internal list of sub-operations
+ List<ColumnOperation> argsAsOperations = new ArrayList<ColumnOperation>();
+ if (columnAdditions != null) {
+ for (HColumnDescriptor newColumn : columnAdditions) {
+ argsAsOperations.add(new AddColumn(master, tableName, newColumn));
+ }
+ }
+ if (columnModifications != null) {
+ for (Pair<byte [], HColumnDescriptor> modColumn : columnModifications) {
+ argsAsOperations.add(new ModifyColumn(
+ master, tableName, modColumn.getFirst(), modColumn.getSecond()));
+ }
+ }
+ if (columnDeletions != null) {
+ for (byte [] columnToReap : columnDeletions) {
+ argsAsOperations.add(new DeleteColumn(master, tableName, columnToReap));
+ }
+ }
+ this.operations = argsAsOperations;
+ }
+
+ @Override
+ protected void postProcess(HRegionInfo hri) throws IOException {
+ // just ask all of the sub-operations to post-process
+ for (ColumnOperation op : operations) {
+ op.postProcess(hri);
+ }
+ }
+
+ @Override
+ protected void updateTableDescriptor(HTableDescriptor desc)
+ throws IOException {
+ // just ask all of the sub-operations to update the descriptor
+ for (ColumnOperation op : operations) {
+ op.updateTableDescriptor(desc);
+ }
+ }
+}
Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/TableOperation.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/TableOperation.java?rev=1181958&r1=1181957&r2=1181958&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/TableOperation.java (original)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/master/TableOperation.java Tue Oct 11 17:44:18 2011
@@ -120,7 +120,8 @@ abstract class TableOperation {
tableExists = true;
if(tableOp instanceof AddColumn || tableOp instanceof ModifyColumn ||
- tableOp instanceof DeleteColumn) {
+ tableOp instanceof DeleteColumn ||
+ tableOp instanceof MultiColumnOperation) {
regionsToProcess.add(info);
}
if (!isBeingServed(serverName) || !isEnabled(info)) {
Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java?rev=1181958&r1=1181957&r2=1181958&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java (original)
+++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java Tue Oct 11 17:44:18 2011
@@ -158,6 +158,7 @@ public class Store implements HeapSize {
this.fs = fs;
this.homedir = getStoreHomedir(basedir, info.getEncodedName(), family.getName());
if (!this.fs.exists(this.homedir)) {
+ LOG.info("No directory exists for family " + family + "; creating one");
if (!this.fs.mkdirs(this.homedir))
throw new IOException("Failed create of: " + this.homedir.toString());
}
Modified: hbase/branches/0.89/src/main/ruby/hbase/admin.rb
URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/ruby/hbase/admin.rb?rev=1181958&r1=1181957&r2=1181958&view=diff
==============================================================================
--- hbase/branches/0.89/src/main/ruby/hbase/admin.rb (original)
+++ hbase/branches/0.89/src/main/ruby/hbase/admin.rb Tue Oct 11 17:44:18 2011
@@ -20,15 +20,17 @@
include Java
+java_import java.util.ArrayList
+
java_import org.apache.hadoop.hbase.client.HBaseAdmin
java_import org.apache.zookeeper.ZooKeeperMain
java_import org.apache.hadoop.hbase.HColumnDescriptor
java_import org.apache.hadoop.hbase.HTableDescriptor
java_import org.apache.hadoop.hbase.io.hfile.Compression
java_import org.apache.hadoop.hbase.regionserver.StoreFile
+java_import org.apache.hadoop.hbase.util.Pair
java_import org.apache.hadoop.hbase.HRegionInfo
java_import org.apache.zookeeper.ZooKeeper
-java_import org.apache.hadoop.hbase.util.Pair
# Wrapper for org.apache.hadoop.hbase.client.HBaseAdmin
@@ -244,6 +246,9 @@ module Hbase
htd = @admin.getTableDescriptor(table_name.to_java_bytes)
# Process all args
+ columnsToAdd = ArrayList.new()
+ columnsToMod = ArrayList.new()
+ columnsToDel = ArrayList.new()
args.each do |arg|
# Normalize args to support column name only alter specs
arg = { NAME => arg } if arg.kind_of?(String)
@@ -258,29 +263,16 @@ module Hbase
# If column already exist, then try to alter it. Create otherwise.
if htd.hasFamily(column_name.to_java_bytes)
- @admin.modifyColumn(table_name, column_name, descriptor)
- if wait == true
- puts "Updating all regions with the new schema..."
- alter_status(table_name)
- end
+ columnsToMod.add(Pair.new(column_name, descriptor))
else
- @admin.addColumn(table_name, descriptor)
- if wait == true
- puts "Updating all regions with the new schema..."
- alter_status(table_name)
- end
+ columnsToAdd.add(descriptor)
end
next
end
-
# Delete column family
if method == "delete"
raise(ArgumentError, "NAME parameter missing for delete method") unless arg[NAME]
- @admin.deleteColumn(table_name, arg[NAME])
- if wait == true
- puts "Updating all regions with the new schema..."
- alter_status(table_name)
- end
+ columnsToDel.add(arg[NAME])
next
end
@@ -299,6 +291,12 @@ module Hbase
# Unknown method
raise ArgumentError, "Unknown method: #{method}"
end
+ # now batch process alter requests
+ @admin.alterTable(table_name, columnsToAdd, columnsToMod, columnsToDel)
+ if wait == true
+ puts "Updating all regions with the new schema..."
+ alter_status(table_name)
+ end
end
def status(format)