You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2020/02/06 15:59:08 UTC
[GitHub] [accumulo] milleruntime opened a new pull request #1501: Use
TabletFile for Tablet, Scans and compaction
milleruntime opened a new pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501
* Made use of TabletFile from Ample through Tablet code, compactions
and scans where object represents a file in a Tablet
* Moved TabletFile to metadata package
* Added TabletFile constructor for the cases when switching volumes and
bulk import where we aren't reading a string from the metadata but have
to create a Path object to resolve the absolute path. That Path object
gets passed to TabletFile so multiple Paths don't need to be created
* Garbage collector will still use strings for delete candidates since
they can be a tablet directory
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r376049619
##########
File path: server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
##########
@@ -247,44 +247,48 @@ public static void splitTablet(KeyExtent extent, Text oldPrevEndRow, double spli
update(context, zooLock, m, extent);
}
- public static void finishSplit(Text metadataEntry, Map<FileRef,DataFileValue> datafileSizes,
- List<FileRef> highDatafilesToRemove, final ServerContext context, ZooLock zooLock) {
+ public static void finishSplit(Text metadataEntry, Map<TabletFile,DataFileValue> datafileSizes,
+ List<TabletFile> highDatafilesToRemove, final ServerContext context, ZooLock zooLock) {
Mutation m = new Mutation(metadataEntry);
TabletsSection.TabletColumnFamily.SPLIT_RATIO_COLUMN.putDelete(m);
TabletsSection.TabletColumnFamily.OLD_PREV_ROW_COLUMN.putDelete(m);
ChoppedColumnFamily.CHOPPED_COLUMN.putDelete(m);
- for (Entry<FileRef,DataFileValue> entry : datafileSizes.entrySet()) {
- m.put(DataFileColumnFamily.NAME, entry.getKey().meta(), new Value(entry.getValue().encode()));
+ for (Entry<TabletFile,DataFileValue> entry : datafileSizes.entrySet()) {
+ m.put(DataFileColumnFamily.NAME, entry.getKey().getMetadataText(),
+ new Value(entry.getValue().encode()));
}
- for (FileRef pathToRemove : highDatafilesToRemove) {
- m.putDelete(DataFileColumnFamily.NAME, pathToRemove.meta());
+ for (TabletFile pathToRemove : highDatafilesToRemove) {
+ m.putDelete(DataFileColumnFamily.NAME, pathToRemove.getMetadataText());
}
update(context, zooLock, m, new KeyExtent(metadataEntry, (Text) null));
}
- public static void finishSplit(KeyExtent extent, Map<FileRef,DataFileValue> datafileSizes,
- List<FileRef> highDatafilesToRemove, ServerContext context, ZooLock zooLock) {
+ public static void finishSplit(KeyExtent extent, Map<TabletFile,DataFileValue> datafileSizes,
+ List<TabletFile> highDatafilesToRemove, ServerContext context, ZooLock zooLock) {
finishSplit(extent.getMetadataEntry(), datafileSizes, highDatafilesToRemove, context, zooLock);
}
- public static void addDeleteEntries(KeyExtent extent, Set<FileRef> datafilesToDelete,
+ /**
+ * datafilesToDelete are strings because they can be a TabletFile or directory
+ */
+ public static void addDeleteEntries(KeyExtent extent, Set<String> datafilesToDelete,
Review comment:
This method was bumping into the GC code that uses tabletDirs. One possible code path for a file to be added to those datafilesToDelete comes from ```GcVolumeUtil.getDeleteTabletOnAllVolumesUri()``` [Here](https://github.com/apache/accumulo/blob/8ea08f8ef59cd199ec23d414d8dbc89be7eb1667/server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java#L627)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on issue #1501: Use TabletFile
for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#issuecomment-583086324
> I finished reviewing, the github feature to mark files as viewed was invaluable for this review. Is the only think using FileRef now write ahead log code?
Thanks for the review.
I don't see where WAL code is using FileRef, maybe you are thinking of LogEntry? LogEntry takes a String for filename, which I think may be another good case for TableFile, if the WAL paths are the same.
FileRef is still used by the two parts I found that use directories - the GC and the old bulkImport CopyFailed FATE operation. I thought of creating an object for TabletDir for these cases but I don't think its worth it (I couldn't come up with a graceful reuse of TabletFile without dropping ```final``` on its members) since I think we can just use Path for these.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r376050741
##########
File path: server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java
##########
@@ -611,12 +609,11 @@ private void deleteTablets(MergeInfo info) throws AccumuloException {
TabletsSection.ServerColumnFamily.TIME_COLUMN.fetch(scanner);
scanner.fetchColumnFamily(DataFileColumnFamily.NAME);
scanner.fetchColumnFamily(TabletsSection.CurrentLocationColumnFamily.NAME);
- Set<FileRef> datafiles = new TreeSet<>();
+ Set<String> datafiles = new TreeSet<>();
for (Entry<Key,Value> entry : scanner) {
Key key = entry.getKey();
if (key.compareColumnFamily(DataFileColumnFamily.NAME) == 0) {
- datafiles
- .add(new FileRef(TabletFileUtil.validate(key.getColumnQualifierData().toString())));
+ datafiles.add(key.getColumnQualifierData().toString());
Review comment:
Hmm this might have been a mistake. I had this code using TabletFile at one point but then had to revert to Strings because of the GC code, per my other comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r376095469
##########
File path: server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java
##########
@@ -611,12 +609,11 @@ private void deleteTablets(MergeInfo info) throws AccumuloException {
TabletsSection.ServerColumnFamily.TIME_COLUMN.fetch(scanner);
scanner.fetchColumnFamily(DataFileColumnFamily.NAME);
scanner.fetchColumnFamily(TabletsSection.CurrentLocationColumnFamily.NAME);
- Set<FileRef> datafiles = new TreeSet<>();
+ Set<String> datafiles = new TreeSet<>();
for (Entry<Key,Value> entry : scanner) {
Key key = entry.getKey();
if (key.compareColumnFamily(DataFileColumnFamily.NAME) == 0) {
- datafiles
- .add(new FileRef(TabletFileUtil.validate(key.getColumnQualifierData().toString())));
+ datafiles.add(key.getColumnQualifierData().toString());
Review comment:
Fixed in 3aacfe603ad9c5d432df1da6411297783adba004
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r375970394
##########
File path: server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
##########
@@ -414,7 +420,7 @@ public static void deleteTable(TableId tableId, boolean insertDeletes, ServerCon
result.addAll(tablet.getLogs());
- tablet.getFilesMap().forEach((tf, v) -> sizes.put(new FileRef(tf.getMetadataEntry()), v));
+ tablet.getFilesMap().forEach(sizes::put);
Review comment:
That is kinda of a neat that `sizes::put` can be done now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r376057065
##########
File path: server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
##########
@@ -247,44 +247,48 @@ public static void splitTablet(KeyExtent extent, Text oldPrevEndRow, double spli
update(context, zooLock, m, extent);
}
- public static void finishSplit(Text metadataEntry, Map<FileRef,DataFileValue> datafileSizes,
- List<FileRef> highDatafilesToRemove, final ServerContext context, ZooLock zooLock) {
+ public static void finishSplit(Text metadataEntry, Map<TabletFile,DataFileValue> datafileSizes,
+ List<TabletFile> highDatafilesToRemove, final ServerContext context, ZooLock zooLock) {
Mutation m = new Mutation(metadataEntry);
TabletsSection.TabletColumnFamily.SPLIT_RATIO_COLUMN.putDelete(m);
TabletsSection.TabletColumnFamily.OLD_PREV_ROW_COLUMN.putDelete(m);
ChoppedColumnFamily.CHOPPED_COLUMN.putDelete(m);
- for (Entry<FileRef,DataFileValue> entry : datafileSizes.entrySet()) {
- m.put(DataFileColumnFamily.NAME, entry.getKey().meta(), new Value(entry.getValue().encode()));
+ for (Entry<TabletFile,DataFileValue> entry : datafileSizes.entrySet()) {
+ m.put(DataFileColumnFamily.NAME, entry.getKey().getMetadataText(),
+ new Value(entry.getValue().encode()));
}
- for (FileRef pathToRemove : highDatafilesToRemove) {
- m.putDelete(DataFileColumnFamily.NAME, pathToRemove.meta());
+ for (TabletFile pathToRemove : highDatafilesToRemove) {
+ m.putDelete(DataFileColumnFamily.NAME, pathToRemove.getMetadataText());
}
update(context, zooLock, m, new KeyExtent(metadataEntry, (Text) null));
}
- public static void finishSplit(KeyExtent extent, Map<FileRef,DataFileValue> datafileSizes,
- List<FileRef> highDatafilesToRemove, ServerContext context, ZooLock zooLock) {
+ public static void finishSplit(KeyExtent extent, Map<TabletFile,DataFileValue> datafileSizes,
+ List<TabletFile> highDatafilesToRemove, ServerContext context, ZooLock zooLock) {
finishSplit(extent.getMetadataEntry(), datafileSizes, highDatafilesToRemove, context, zooLock);
}
- public static void addDeleteEntries(KeyExtent extent, Set<FileRef> datafilesToDelete,
+ /**
+ * datafilesToDelete are strings because they can be a TabletFile or directory
+ */
+ public static void addDeleteEntries(KeyExtent extent, Set<String> datafilesToDelete,
Review comment:
Thanks for the pointer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r376037113
##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/compaction/MajorCompactionRequest.java
##########
@@ -105,7 +105,7 @@ public MajorCompactionReason getReason() {
return reason;
}
- public Map<FileRef,DataFileValue> getFiles() {
+ public Map<TabletFile,DataFileValue> getFiles() {
Review comment:
This class is used by CompactionStrategy which is a pluggable component. So this changes could break existing compaction strategy plugins.
I think this change is ok because with these types being outside of SPI, its hard to analyze in a rigorous way. It would not surprise me if there were already other breaking changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r375972940
##########
File path: server/base/src/test/java/org/apache/accumulo/server/util/FileUtilTest.java
##########
@@ -66,13 +66,13 @@ public void createTmpDir() throws IOException {
@Test
public void testToPathStrings() {
- Collection<FileRef> c = new java.util.ArrayList<>();
- FileRef r1 = createMock(FileRef.class);
- expect(r1.path()).andReturn(new Path("/foo"));
+ Collection<TabletFile> c = new java.util.ArrayList<>();
+ TabletFile r1 = createMock(TabletFile.class);
+ expect(r1.getNormalizedPath()).andReturn("/foo");
replay(r1);
c.add(r1);
- FileRef r2 = createMock(FileRef.class);
- expect(r2.path()).andReturn(new Path("/bar"));
+ TabletFile r2 = createMock(TabletFile.class);
Review comment:
This test seems suspicious. I am wondering if this test is still needed. I will look into this later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r376054859
##########
File path: server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java
##########
@@ -159,14 +160,14 @@ public static TabletFiles updateTabletVolumes(ServerContext context, ZooLock zoo
}
}
- for (Entry<FileRef,DataFileValue> entry : tabletFiles.datafiles.entrySet()) {
- String metaPath = entry.getKey().meta().toString();
+ for (Entry<TabletFile,DataFileValue> entry : tabletFiles.datafiles.entrySet()) {
+ String metaPath = entry.getKey().getNormalizedPath();
Path switchedPath = switchVolume(metaPath, FileType.TABLE, replacements);
if (switchedPath != null) {
filesToRemove.add(entry.getKey());
- FileRef switchedRef = new FileRef(switchedPath.toString(), switchedPath);
- filesToAdd.put(switchedRef, entry.getValue());
- ret.datafiles.put(switchedRef, entry.getValue());
+ TabletFile switchedFile = new TabletFile(switchedPath, switchedPath.toString());
Review comment:
> but I don't see where it will write out the new path. Is this done during compaction?
I am not sure, was there a specific class you were looking at? I can take a look.
> If so, then we will have problems if we throw an exception since the string won't ever exist in the metadata.
It would be annoying if making TabletFile more strict causes code that is working for the wrong reasons to break, but it would be nice to cause code that is not working but failing silently to break.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on issue #1501: Use TabletFile
for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on issue #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#issuecomment-583087571
> I don't see where WAL code is using FileRef, maybe you are thinking of LogEntry?
I was just guessing at what was using it. Mainly I was curious FileRef was not deleted, I should have just asked that. You already answered that question.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r376046946
##########
File path: server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java
##########
@@ -159,14 +160,14 @@ public static TabletFiles updateTabletVolumes(ServerContext context, ZooLock zoo
}
}
- for (Entry<FileRef,DataFileValue> entry : tabletFiles.datafiles.entrySet()) {
- String metaPath = entry.getKey().meta().toString();
+ for (Entry<TabletFile,DataFileValue> entry : tabletFiles.datafiles.entrySet()) {
+ String metaPath = entry.getKey().getNormalizedPath();
Path switchedPath = switchVolume(metaPath, FileType.TABLE, replacements);
if (switchedPath != null) {
filesToRemove.add(entry.getKey());
- FileRef switchedRef = new FileRef(switchedPath.toString(), switchedPath);
- filesToAdd.put(switchedRef, entry.getValue());
- ret.datafiles.put(switchedRef, entry.getValue());
+ TabletFile switchedFile = new TabletFile(switchedPath, switchedPath.toString());
Review comment:
That is a good point. The volume replacement code (which is where this is used) will delete the original file but I don't see where it will write out the new path. Is this done during compaction? If so, then we will have problems if we throw an exception since the string won't ever exist in the metadata.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r375972940
##########
File path: server/base/src/test/java/org/apache/accumulo/server/util/FileUtilTest.java
##########
@@ -66,13 +66,13 @@ public void createTmpDir() throws IOException {
@Test
public void testToPathStrings() {
- Collection<FileRef> c = new java.util.ArrayList<>();
- FileRef r1 = createMock(FileRef.class);
- expect(r1.path()).andReturn(new Path("/foo"));
+ Collection<TabletFile> c = new java.util.ArrayList<>();
+ TabletFile r1 = createMock(TabletFile.class);
+ expect(r1.getNormalizedPath()).andReturn("/foo");
replay(r1);
c.add(r1);
- FileRef r2 = createMock(FileRef.class);
- expect(r2.path()).andReturn(new Path("/bar"));
+ TabletFile r2 = createMock(TabletFile.class);
Review comment:
This test seems suspicious. I am wondering if its still needed. I will look into this later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r375964924
##########
File path: server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java
##########
@@ -159,14 +160,14 @@ public static TabletFiles updateTabletVolumes(ServerContext context, ZooLock zoo
}
}
- for (Entry<FileRef,DataFileValue> entry : tabletFiles.datafiles.entrySet()) {
- String metaPath = entry.getKey().meta().toString();
+ for (Entry<TabletFile,DataFileValue> entry : tabletFiles.datafiles.entrySet()) {
+ String metaPath = entry.getKey().getNormalizedPath();
Path switchedPath = switchVolume(metaPath, FileType.TABLE, replacements);
if (switchedPath != null) {
filesToRemove.add(entry.getKey());
- FileRef switchedRef = new FileRef(switchedPath.toString(), switchedPath);
- filesToAdd.put(switchedRef, entry.getValue());
- ret.datafiles.put(switchedRef, entry.getValue());
+ TabletFile switchedFile = new TabletFile(switchedPath, switchedPath.toString());
Review comment:
When I looked at this code I found myself wondering if any follow on code relies on the metadata entry for the TabletFile created here. If so that could result in a silent bug (where we try to delete something in the metadata table that does not exists). This makes me think it would be nice to be able to construct a TabletFile w/o a metadata entry and anything trying to get the metadata path would get an exception. This would cause things to fail fast.
Could possibly have a TabletFile constructor that only takes a path and when this constructor is used to create a the object, attempts to get the metadata entry result in an exception.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r376055669
##########
File path: server/base/src/test/java/org/apache/accumulo/server/util/FileUtilTest.java
##########
@@ -66,13 +66,13 @@ public void createTmpDir() throws IOException {
@Test
public void testToPathStrings() {
- Collection<FileRef> c = new java.util.ArrayList<>();
- FileRef r1 = createMock(FileRef.class);
- expect(r1.path()).andReturn(new Path("/foo"));
+ Collection<TabletFile> c = new java.util.ArrayList<>();
+ TabletFile r1 = createMock(TabletFile.class);
+ expect(r1.getNormalizedPath()).andReturn("/foo");
replay(r1);
c.add(r1);
- FileRef r2 = createMock(FileRef.class);
- expect(r2.path()).andReturn(new Path("/bar"));
+ TabletFile r2 = createMock(TabletFile.class);
Review comment:
I think this test is ok, its just making sure that a method in FileUtil calls a very specific method on TabletFile. I think the method it is testing could be replaced with a one line stream operation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r375969773
##########
File path: server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
##########
@@ -247,44 +247,48 @@ public static void splitTablet(KeyExtent extent, Text oldPrevEndRow, double spli
update(context, zooLock, m, extent);
}
- public static void finishSplit(Text metadataEntry, Map<FileRef,DataFileValue> datafileSizes,
- List<FileRef> highDatafilesToRemove, final ServerContext context, ZooLock zooLock) {
+ public static void finishSplit(Text metadataEntry, Map<TabletFile,DataFileValue> datafileSizes,
+ List<TabletFile> highDatafilesToRemove, final ServerContext context, ZooLock zooLock) {
Mutation m = new Mutation(metadataEntry);
TabletsSection.TabletColumnFamily.SPLIT_RATIO_COLUMN.putDelete(m);
TabletsSection.TabletColumnFamily.OLD_PREV_ROW_COLUMN.putDelete(m);
ChoppedColumnFamily.CHOPPED_COLUMN.putDelete(m);
- for (Entry<FileRef,DataFileValue> entry : datafileSizes.entrySet()) {
- m.put(DataFileColumnFamily.NAME, entry.getKey().meta(), new Value(entry.getValue().encode()));
+ for (Entry<TabletFile,DataFileValue> entry : datafileSizes.entrySet()) {
+ m.put(DataFileColumnFamily.NAME, entry.getKey().getMetadataText(),
+ new Value(entry.getValue().encode()));
}
- for (FileRef pathToRemove : highDatafilesToRemove) {
- m.putDelete(DataFileColumnFamily.NAME, pathToRemove.meta());
+ for (TabletFile pathToRemove : highDatafilesToRemove) {
+ m.putDelete(DataFileColumnFamily.NAME, pathToRemove.getMetadataText());
}
update(context, zooLock, m, new KeyExtent(metadataEntry, (Text) null));
}
- public static void finishSplit(KeyExtent extent, Map<FileRef,DataFileValue> datafileSizes,
- List<FileRef> highDatafilesToRemove, ServerContext context, ZooLock zooLock) {
+ public static void finishSplit(KeyExtent extent, Map<TabletFile,DataFileValue> datafileSizes,
+ List<TabletFile> highDatafilesToRemove, ServerContext context, ZooLock zooLock) {
finishSplit(extent.getMetadataEntry(), datafileSizes, highDatafilesToRemove, context, zooLock);
}
- public static void addDeleteEntries(KeyExtent extent, Set<FileRef> datafilesToDelete,
+ /**
+ * datafilesToDelete are strings because they can be a TabletFile or directory
+ */
+ public static void addDeleteEntries(KeyExtent extent, Set<String> datafilesToDelete,
Review comment:
Why did this end up `Set<String>`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r377342835
##########
File path: server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java
##########
@@ -159,14 +160,14 @@ public static TabletFiles updateTabletVolumes(ServerContext context, ZooLock zoo
}
}
- for (Entry<FileRef,DataFileValue> entry : tabletFiles.datafiles.entrySet()) {
- String metaPath = entry.getKey().meta().toString();
+ for (Entry<TabletFile,DataFileValue> entry : tabletFiles.datafiles.entrySet()) {
+ String metaPath = entry.getKey().getNormalizedPath();
Path switchedPath = switchVolume(metaPath, FileType.TABLE, replacements);
if (switchedPath != null) {
filesToRemove.add(entry.getKey());
- FileRef switchedRef = new FileRef(switchedPath.toString(), switchedPath);
- filesToAdd.put(switchedRef, entry.getValue());
- ret.datafiles.put(switchedRef, entry.getValue());
+ TabletFile switchedFile = new TabletFile(switchedPath, switchedPath.toString());
Review comment:
I was looking at the code and it seems like there may be an existing bug. I opened #1501
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime merged pull request #1501: Use TabletFile
for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
milleruntime merged pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r375974051
##########
File path: server/master/src/main/java/org/apache/accumulo/master/TabletGroupWatcher.java
##########
@@ -611,12 +609,11 @@ private void deleteTablets(MergeInfo info) throws AccumuloException {
TabletsSection.ServerColumnFamily.TIME_COLUMN.fetch(scanner);
scanner.fetchColumnFamily(DataFileColumnFamily.NAME);
scanner.fetchColumnFamily(TabletsSection.CurrentLocationColumnFamily.NAME);
- Set<FileRef> datafiles = new TreeSet<>();
+ Set<String> datafiles = new TreeSet<>();
for (Entry<Key,Value> entry : scanner) {
Key key = entry.getKey();
if (key.compareColumnFamily(DataFileColumnFamily.NAME) == 0) {
- datafiles
- .add(new FileRef(TabletFileUtil.validate(key.getColumnQualifierData().toString())));
+ datafiles.add(key.getColumnQualifierData().toString());
Review comment:
Why drop validation here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner edited a comment on issue #1501: Use
TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
keith-turner edited a comment on issue #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#issuecomment-583087571
> I don't see where WAL code is using FileRef, maybe you are thinking of LogEntry?
I was just guessing at what was using it. Mainly I was curious why FileRef was not deleted, I should have just asked that. You already answered that question.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1501: Use TabletFile for Tablet, Scans and compaction
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1501: Use TabletFile for Tablet, Scans and compaction
URL: https://github.com/apache/accumulo/pull/1501#discussion_r376056967
##########
File path: server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java
##########
@@ -159,14 +160,14 @@ public static TabletFiles updateTabletVolumes(ServerContext context, ZooLock zoo
}
}
- for (Entry<FileRef,DataFileValue> entry : tabletFiles.datafiles.entrySet()) {
- String metaPath = entry.getKey().meta().toString();
+ for (Entry<TabletFile,DataFileValue> entry : tabletFiles.datafiles.entrySet()) {
+ String metaPath = entry.getKey().getNormalizedPath();
Path switchedPath = switchVolume(metaPath, FileType.TABLE, replacements);
if (switchedPath != null) {
filesToRemove.add(entry.getKey());
- FileRef switchedRef = new FileRef(switchedPath.toString(), switchedPath);
- filesToAdd.put(switchedRef, entry.getValue());
- ret.datafiles.put(switchedRef, entry.getValue());
+ TabletFile switchedFile = new TabletFile(switchedPath, switchedPath.toString());
Review comment:
> > but I don't see where it will write out the new path. Is this done during compaction?
>
> I am not sure, was there a specific class you were looking at? I can take a look.
Just this method, ```VolumeUtil.updateTabletVolumes()``` and the only place its called, [Tablet](https://github.com/apache/accumulo/blob/8ea08f8ef59cd199ec23d414d8dbc89be7eb1667/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java#L342).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services