You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2020/01/09 18:09:16 UTC
[GitHub] [accumulo] milleruntime opened a new pull request #1471: Create
TabletFile for use in TabletMetadata. Closes #1396
milleruntime opened a new pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471
* Created new TabletFile object as a strong type to replace where
strings are used to reference metadata tablet file paths
* Drop parts of code dealing with relative paths
* TabletData only uses TabletFile to create a FileRef for now, follow on
work can be done to utilize TabletFile in place of FileRef
* Update tests to use absolute paths
* Add new Translator for TabletFile
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r365396618
##########
File path: core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletFile.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import java.util.Objects;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+
+/**
+ * Object representing a tablet file entry in the metadata table. Keeps a string of the exact entry
+ * of what is in the metadata table for the column qualifier of the
+ * {@link org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily}
+ *
+ * As of 2.1, Tablet file paths should now be only absolute URIs with the removal of relative paths
+ * in Upgrader9to10.upgradeRelativePaths()
+ */
+public class TabletFile implements Comparable<TabletFile> {
+ // parts of an absolute URI, like "hdfs://1.2.3.4/accumulo/tables/2a/t-0003/C0004.rf"
+ private final String volume; // hdfs://1.2.3.4/accumulo
+ private final TableId tableId; // 2a
+ private final String tabletDir; // t-0003
+ private final String fileName; // C0004.rf
+ private final String metadataEntry;
+ private final String normalizedPath; // 2a/t-0003/C0004.rf
+
+ public TabletFile(String metadataEntry) {
+ this.metadataEntry = Objects.requireNonNull(metadataEntry);
+ String errorMsg = " is missing from tablet file metadata entry: " + metadataEntry;
+
+ Path metaPath = new Path(metadataEntry);
+
+ // use Path object to step backwards from the filename through all the parts
+ this.fileName = metaPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(fileName);
+
+ Path tabletDirPath = Objects.requireNonNull(metaPath.getParent(), "Tablet dir" + errorMsg);
+ this.tabletDir = tabletDirPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(tabletDir);
+
+ Path tableIdPath = Objects.requireNonNull(tabletDirPath.getParent(), "Table ID" + errorMsg);
+ this.tableId = TableId.of(tableIdPath.getName());
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(tableId.canonical());
+
+ Path volumePath = Objects.requireNonNull(
Review comment:
Could do the following which avoid calling `TabletFileUtil.getVolumeFromFullPath`. This method converts a path to a string internally and then back to a path. The following code uses the exisitng path object.
```java
Path tablePath = Objects.requireNonNull(tableIdPath.getParent(), "Table" + errorMsg);
Preconditions.checkArgument(tablePath.getName().equals(Constants.HDFS_TABLES_DIR));
Path volumePath = Objects.requireNonNull(tablePath.getParent());
```
When profiling in the past I have seen the Hadoop Path object show up. So if its easy to avoid creating one, I try to do that. This code would be used pretty frequently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r365399284
##########
File path: core/src/test/java/org/apache/accumulo/core/metadata/schema/TabletFileTest.java
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import org.apache.accumulo.core.data.TableId;
+import org.junit.Test;
+
+public class TabletFileTest {
+
+ private void test(String metadataEntry, TableId tableId, String tabletDir, String fileName) {
+ TabletFile tabletFile = new TabletFile(metadataEntry);
+
+ assertEquals(metadataEntry, tabletFile.getMetadataEntry());
+ assertEquals(tableId, tabletFile.getTableId());
+ assertEquals(tabletDir, tabletFile.getTabletDir());
+ assertEquals(fileName, tabletFile.getFileName());
+
Review comment:
Would be nice to also verify normalization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r365392353
##########
File path: core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletFile.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import java.util.Objects;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+
+/**
+ * Object representing a tablet file entry in the metadata table. Keeps a string of the exact entry
+ * of what is in the metadata table for the column qualifier of the
+ * {@link org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily}
+ *
+ * As of 2.1, Tablet file paths should now be only absolute URIs with the removal of relative paths
Review comment:
Not complete sure, but I think unless a `<p>` this will not show up as its own paragraph in the rendered javadoc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r366549747
##########
File path: core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletFile.java
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import java.util.Objects;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Object representing a tablet file entry in the metadata table. Keeps a string of the exact entry
+ * of what is in the metadata table for the column qualifier of the
+ * {@link org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily}
+ * <p>
+ * As of 2.1, Tablet file paths should now be only absolute URIs with the removal of relative paths
+ * in Upgrader9to10.upgradeRelativePaths()
+ */
+public class TabletFile implements Comparable<TabletFile> {
+ // parts of an absolute URI, like "hdfs://1.2.3.4/accumulo/tables/2a/t-0003/C0004.rf"
+ private final String volume; // hdfs://1.2.3.4/accumulo
+ private final TableId tableId; // 2a
+ private final String tabletDir; // t-0003
+ private final String fileName; // C0004.rf
+ private final String metadataEntry;
+ private final Path metaPath;
+ private final String normalizedPath;
+
+ public TabletFile(String metadataEntry) {
+ this.metadataEntry = Objects.requireNonNull(metadataEntry);
+ String errorMsg = " is missing/invalid from tablet file metadata entry: " + metadataEntry;
+
+ this.metaPath = new Path(metadataEntry);
+
+ // use Path object to step backwards from the filename through all the parts
+ this.fileName = metaPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(fileName);
+
+ Path tabletDirPath = Objects.requireNonNull(metaPath.getParent(), "Tablet dir" + errorMsg);
Review comment:
Good point. I think I'd rather make one generic error message that includes the metadataEntry than trying to describe the part of the string which had an error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime merged pull request #1471: Create
TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
milleruntime merged pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r365954495
##########
File path: core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletFile.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import java.util.Objects;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+
+/**
+ * Object representing a tablet file entry in the metadata table. Keeps a string of the exact entry
+ * of what is in the metadata table for the column qualifier of the
+ * {@link org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily}
+ *
+ * As of 2.1, Tablet file paths should now be only absolute URIs with the removal of relative paths
+ * in Upgrader9to10.upgradeRelativePaths()
+ */
+public class TabletFile implements Comparable<TabletFile> {
+ // parts of an absolute URI, like "hdfs://1.2.3.4/accumulo/tables/2a/t-0003/C0004.rf"
+ private final String volume; // hdfs://1.2.3.4/accumulo
+ private final TableId tableId; // 2a
+ private final String tabletDir; // t-0003
+ private final String fileName; // C0004.rf
+ private final String metadataEntry;
+ private final String normalizedPath; // 2a/t-0003/C0004.rf
+
+ public TabletFile(String metadataEntry) {
+ this.metadataEntry = Objects.requireNonNull(metadataEntry);
+ String errorMsg = " is missing from tablet file metadata entry: " + metadataEntry;
+
+ Path metaPath = new Path(metadataEntry);
+
+ // use Path object to step backwards from the filename through all the parts
+ this.fileName = metaPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(fileName);
+
+ Path tabletDirPath = Objects.requireNonNull(metaPath.getParent(), "Tablet dir" + errorMsg);
+ this.tabletDir = tabletDirPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(tabletDir);
+
+ Path tableIdPath = Objects.requireNonNull(tabletDirPath.getParent(), "Table ID" + errorMsg);
+ this.tableId = TableId.of(tableIdPath.getName());
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(tableId.canonical());
+
+ Path volumePath = Objects.requireNonNull(
Review comment:
I like this a lot better. I can then drop TabletFileUtil, since this was the last thing it was doing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r365400886
##########
File path: core/src/test/java/org/apache/accumulo/core/metadata/schema/TabletFileTest.java
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import org.apache.accumulo.core.data.TableId;
+import org.junit.Test;
+
+public class TabletFileTest {
+
+ private void test(String metadataEntry, TableId tableId, String tabletDir, String fileName) {
+ TabletFile tabletFile = new TabletFile(metadataEntry);
+
+ assertEquals(metadataEntry, tabletFile.getMetadataEntry());
+ assertEquals(tableId, tabletFile.getTableId());
+ assertEquals(tabletDir, tabletFile.getTabletDir());
+ assertEquals(fileName, tabletFile.getFileName());
+
+ }
+
+ @Test
+ public void testValidPaths() {
+ test("hdfs://localhost:8020/accumulo/tables/2a/default_tablet/F0000070.rf", TableId.of("2a"),
+ "default_tablet", "F0000070.rf");
+ test("hdfs://nn1:9000/accumulo/tables/5a/t-0005/C0009.rf", TableId.of("5a"), "t-0005",
+ "C0009.rf");
+ }
+
+ @Test
+ public void testBadPaths() {
Review comment:
The following are some additional bad paths that would be nice to check.
`hdfs://localhost:8020/accumulo/tablets/2a/default_tablet/F0000070.rf`
`hdfs://localhost:8020/accumulo/2a/default_tablet/F0000070.rf`
`/accumulo/tables/2a/default_tablet/F0000070.rf`
`hdfs://localhost:8020/accumulo/tables/2a/F0000070.rf`
`hdfs://localhost:8020/accumulo/tables/F0000070.rf`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r366050529
##########
File path: core/src/test/java/org/apache/accumulo/core/metadata/schema/TabletFileTest.java
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import org.apache.accumulo.core.data.TableId;
+import org.junit.Test;
+
+public class TabletFileTest {
+
+ private void test(String metadataEntry, TableId tableId, String tabletDir, String fileName) {
+ TabletFile tabletFile = new TabletFile(metadataEntry);
+
+ assertEquals(metadataEntry, tabletFile.getMetadataEntry());
+ assertEquals(tableId, tabletFile.getTableId());
+ assertEquals(tabletDir, tabletFile.getTabletDir());
+ assertEquals(fileName, tabletFile.getFileName());
+
+ }
+
+ @Test
+ public void testValidPaths() {
+ test("hdfs://localhost:8020/accumulo/tables/2a/default_tablet/F0000070.rf", TableId.of("2a"),
+ "default_tablet", "F0000070.rf");
+ test("hdfs://nn1:9000/accumulo/tables/5a/t-0005/C0009.rf", TableId.of("5a"), "t-0005",
+ "C0009.rf");
+ }
+
+ @Test
+ public void testBadPaths() {
Review comment:
Maybe ensure that `volumePath.toUri().getScheme()` is not null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] ctubbsii commented on a change in pull request #1471:
Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r365483245
##########
File path: core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletFile.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import java.util.Objects;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+
+/**
+ * Object representing a tablet file entry in the metadata table. Keeps a string of the exact entry
+ * of what is in the metadata table for the column qualifier of the
+ * {@link org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily}
+ *
+ * As of 2.1, Tablet file paths should now be only absolute URIs with the removal of relative paths
+ * in Upgrader9to10.upgradeRelativePaths()
+ */
+public class TabletFile implements Comparable<TabletFile> {
+ // parts of an absolute URI, like "hdfs://1.2.3.4/accumulo/tables/2a/t-0003/C0004.rf"
+ private final String volume; // hdfs://1.2.3.4/accumulo
+ private final TableId tableId; // 2a
+ private final String tabletDir; // t-0003
+ private final String fileName; // C0004.rf
+ private final String metadataEntry;
+ private final String normalizedPath; // 2a/t-0003/C0004.rf
+
+ public TabletFile(String metadataEntry) {
+ this.metadataEntry = Objects.requireNonNull(metadataEntry);
+ String errorMsg = " is missing from tablet file metadata entry: " + metadataEntry;
+
+ Path metaPath = new Path(metadataEntry);
+
+ // use Path object to step backwards from the filename through all the parts
+ this.fileName = metaPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(fileName);
+
+ Path tabletDirPath = Objects.requireNonNull(metaPath.getParent(), "Tablet dir" + errorMsg);
+ this.tabletDir = tabletDirPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(tabletDir);
+
+ Path tableIdPath = Objects.requireNonNull(tabletDirPath.getParent(), "Table ID" + errorMsg);
+ this.tableId = TableId.of(tableIdPath.getName());
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(tableId.canonical());
+
+ Path volumePath = Objects.requireNonNull(
+ TabletFileUtil.getVolumeFromFullPath(metaPath, "tables"), "Volume" + errorMsg);
+ this.volume = volumePath.toString();
+
+ this.normalizedPath = volume + Constants.HDFS_TABLES_DIR + "/" + tableId.canonical() + "/"
Review comment:
I can attest to that. I had the same question on an earlier PR, and Mike and I tested all sorts of combinations together as a sanity check to be sure that it does... at least with the minimum Hadoop version we're specifying in our pom.xml.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r365398220
##########
File path: core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletFile.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import java.util.Objects;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+
+/**
+ * Object representing a tablet file entry in the metadata table. Keeps a string of the exact entry
+ * of what is in the metadata table for the column qualifier of the
+ * {@link org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily}
+ *
+ * As of 2.1, Tablet file paths should now be only absolute URIs with the removal of relative paths
+ * in Upgrader9to10.upgradeRelativePaths()
+ */
+public class TabletFile implements Comparable<TabletFile> {
+ // parts of an absolute URI, like "hdfs://1.2.3.4/accumulo/tables/2a/t-0003/C0004.rf"
+ private final String volume; // hdfs://1.2.3.4/accumulo
+ private final TableId tableId; // 2a
+ private final String tabletDir; // t-0003
+ private final String fileName; // C0004.rf
+ private final String metadataEntry;
+ private final String normalizedPath; // 2a/t-0003/C0004.rf
+
+ public TabletFile(String metadataEntry) {
+ this.metadataEntry = Objects.requireNonNull(metadataEntry);
+ String errorMsg = " is missing from tablet file metadata entry: " + metadataEntry;
+
+ Path metaPath = new Path(metadataEntry);
+
+ // use Path object to step backwards from the filename through all the parts
+ this.fileName = metaPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(fileName);
+
+ Path tabletDirPath = Objects.requireNonNull(metaPath.getParent(), "Tablet dir" + errorMsg);
+ this.tabletDir = tabletDirPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(tabletDir);
+
+ Path tableIdPath = Objects.requireNonNull(tabletDirPath.getParent(), "Table ID" + errorMsg);
+ this.tableId = TableId.of(tableIdPath.getName());
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(tableId.canonical());
+
+ Path volumePath = Objects.requireNonNull(
+ TabletFileUtil.getVolumeFromFullPath(metaPath, "tables"), "Volume" + errorMsg);
+ this.volume = volumePath.toString();
+
+ this.normalizedPath = volume + Constants.HDFS_TABLES_DIR + "/" + tableId.canonical() + "/"
Review comment:
Do you know if Hadoop's path object normalizes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r366011583
##########
File path: core/src/test/java/org/apache/accumulo/core/metadata/schema/TabletFileTest.java
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import org.apache.accumulo.core.data.TableId;
+import org.junit.Test;
+
+public class TabletFileTest {
+
+ private void test(String metadataEntry, TableId tableId, String tabletDir, String fileName) {
+ TabletFile tabletFile = new TabletFile(metadataEntry);
+
+ assertEquals(metadataEntry, tabletFile.getMetadataEntry());
+ assertEquals(tableId, tabletFile.getTableId());
+ assertEquals(tabletDir, tabletFile.getTabletDir());
+ assertEquals(fileName, tabletFile.getFileName());
+
+ }
+
+ @Test
+ public void testValidPaths() {
+ test("hdfs://localhost:8020/accumulo/tables/2a/default_tablet/F0000070.rf", TableId.of("2a"),
+ "default_tablet", "F0000070.rf");
+ test("hdfs://nn1:9000/accumulo/tables/5a/t-0005/C0009.rf", TableId.of("5a"), "t-0005",
+ "C0009.rf");
+ }
+
+ @Test
+ public void testBadPaths() {
Review comment:
I just tested these and noticed we need more validation on the volume. Since we are traversing backwards up from the filename, anything before "tables" will be a valid volume. Any ideas? The best I can come up with is what I've seen... valid if ```path.contains(":")```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r365413876
##########
File path: core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletFile.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import java.util.Objects;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+
+/**
+ * Object representing a tablet file entry in the metadata table. Keeps a string of the exact entry
+ * of what is in the metadata table for the column qualifier of the
+ * {@link org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily}
+ *
+ * As of 2.1, Tablet file paths should now be only absolute URIs with the removal of relative paths
+ * in Upgrader9to10.upgradeRelativePaths()
+ */
+public class TabletFile implements Comparable<TabletFile> {
+ // parts of an absolute URI, like "hdfs://1.2.3.4/accumulo/tables/2a/t-0003/C0004.rf"
+ private final String volume; // hdfs://1.2.3.4/accumulo
+ private final TableId tableId; // 2a
+ private final String tabletDir; // t-0003
+ private final String fileName; // C0004.rf
+ private final String metadataEntry;
+ private final String normalizedPath; // 2a/t-0003/C0004.rf
+
+ public TabletFile(String metadataEntry) {
+ this.metadataEntry = Objects.requireNonNull(metadataEntry);
+ String errorMsg = " is missing from tablet file metadata entry: " + metadataEntry;
+
+ Path metaPath = new Path(metadataEntry);
+
+ // use Path object to step backwards from the filename through all the parts
+ this.fileName = metaPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(fileName);
+
+ Path tabletDirPath = Objects.requireNonNull(metaPath.getParent(), "Tablet dir" + errorMsg);
+ this.tabletDir = tabletDirPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(tabletDir);
+
+ Path tableIdPath = Objects.requireNonNull(tabletDirPath.getParent(), "Table ID" + errorMsg);
+ this.tableId = TableId.of(tableIdPath.getName());
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(tableId.canonical());
+
+ Path volumePath = Objects.requireNonNull(
+ TabletFileUtil.getVolumeFromFullPath(metaPath, "tables"), "Volume" + errorMsg);
+ this.volume = volumePath.toString();
+
+ this.normalizedPath = volume + Constants.HDFS_TABLES_DIR + "/" + tableId.canonical() + "/"
Review comment:
It does, which I tested with my previous recent change to ```Upgrader9to10Test```.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] ctubbsii commented on a change in pull request #1471:
Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r365482991
##########
File path: core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletFile.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import java.util.Objects;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+
+/**
+ * Object representing a tablet file entry in the metadata table. Keeps a string of the exact entry
+ * of what is in the metadata table for the column qualifier of the
+ * {@link org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily}
+ *
+ * As of 2.1, Tablet file paths should now be only absolute URIs with the removal of relative paths
Review comment:
True.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r365414262
##########
File path: core/src/test/java/org/apache/accumulo/core/metadata/schema/TabletFileTest.java
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import org.apache.accumulo.core.data.TableId;
+import org.junit.Test;
+
+public class TabletFileTest {
+
+ private void test(String metadataEntry, TableId tableId, String tabletDir, String fileName) {
+ TabletFile tabletFile = new TabletFile(metadataEntry);
+
+ assertEquals(metadataEntry, tabletFile.getMetadataEntry());
+ assertEquals(tableId, tabletFile.getTableId());
+ assertEquals(tabletDir, tabletFile.getTabletDir());
+ assertEquals(fileName, tabletFile.getFileName());
+
Review comment:
I will add more tests with your ideas. I thought I had a normalize test but I was thinking of the recent one I added for the upgrade code [here](https://github.com/apache/accumulo/blob/ad107377bf2f2e4751b0b8ebe6c2942a3f078de8/server/master/src/test/java/org/apache/accumulo/master/upgrade/Upgrader9to10Test.java#L247).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r366538530
##########
File path: core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletFile.java
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import java.util.Objects;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.data.TableId;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Object representing a tablet file entry in the metadata table. Keeps a string of the exact entry
+ * of what is in the metadata table for the column qualifier of the
+ * {@link org.apache.accumulo.core.metadata.schema.MetadataSchema.TabletsSection.DataFileColumnFamily}
+ * <p>
+ * As of 2.1, Tablet file paths should now be only absolute URIs with the removal of relative paths
+ * in Upgrader9to10.upgradeRelativePaths()
+ */
+public class TabletFile implements Comparable<TabletFile> {
+ // parts of an absolute URI, like "hdfs://1.2.3.4/accumulo/tables/2a/t-0003/C0004.rf"
+ private final String volume; // hdfs://1.2.3.4/accumulo
+ private final TableId tableId; // 2a
+ private final String tabletDir; // t-0003
+ private final String fileName; // C0004.rf
+ private final String metadataEntry;
+ private final Path metaPath;
+ private final String normalizedPath;
+
+ public TabletFile(String metadataEntry) {
+ this.metadataEntry = Objects.requireNonNull(metadataEntry);
+ String errorMsg = " is missing/invalid from tablet file metadata entry: " + metadataEntry;
+
+ this.metaPath = new Path(metadataEntry);
+
+ // use Path object to step backwards from the filename through all the parts
+ this.fileName = metaPath.getName();
+ MetadataSchema.TabletsSection.ServerColumnFamily.validateDirCol(fileName);
+
+ Path tabletDirPath = Objects.requireNonNull(metaPath.getParent(), "Tablet dir" + errorMsg);
Review comment:
There is a lot of string concatenation (which causes object allocation) for error messages that will usually never be seen in this method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] milleruntime commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r366392699
##########
File path: core/src/test/java/org/apache/accumulo/core/metadata/schema/TabletFileTest.java
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import org.apache.accumulo.core.data.TableId;
+import org.junit.Test;
+
+public class TabletFileTest {
+
+ private void test(String metadataEntry, TableId tableId, String tabletDir, String fileName) {
+ TabletFile tabletFile = new TabletFile(metadataEntry);
+
+ assertEquals(metadataEntry, tabletFile.getMetadataEntry());
+ assertEquals(tableId, tabletFile.getTableId());
+ assertEquals(tabletDir, tabletFile.getTabletDir());
+ assertEquals(fileName, tabletFile.getFileName());
+
+ }
+
+ @Test
+ public void testValidPaths() {
+ test("hdfs://localhost:8020/accumulo/tables/2a/default_tablet/F0000070.rf", TableId.of("2a"),
+ "default_tablet", "F0000070.rf");
+ test("hdfs://nn1:9000/accumulo/tables/5a/t-0005/C0009.rf", TableId.of("5a"), "t-0005",
+ "C0009.rf");
+ }
+
+ @Test
+ public void testBadPaths() {
Review comment:
That works great thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [accumulo] keith-turner commented on a change in pull request
#1471: Create TabletFile for use in TabletMetadata. Closes #1396
Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1471: Create TabletFile for use in TabletMetadata. Closes #1396
URL: https://github.com/apache/accumulo/pull/1471#discussion_r366542217
##########
File path: core/src/test/java/org/apache/accumulo/core/metadata/schema/TabletFileTest.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.accumulo.core.metadata.schema;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import org.apache.accumulo.core.data.TableId;
+import org.junit.Test;
+
+public class TabletFileTest {
+
+ private void test(String metadataEntry, String volume, String tableId, String tabletDir,
+ String fileName) {
+ TabletFile tabletFile = new TabletFile(metadataEntry);
+
+ assertEquals(volume, tabletFile.getVolume());
+ assertEquals(metadataEntry, tabletFile.getMetadataEntry());
+ assertEquals(TableId.of(tableId), tabletFile.getTableId());
+ assertEquals(tabletDir, tabletFile.getTabletDir());
+ assertEquals(fileName, tabletFile.getFileName());
+ }
+
+ @Test
+ public void testValidPaths() {
+ test("hdfs://localhost:8020/accumulo/tables/2a/default_tablet/F0000070.rf",
+ "hdfs://localhost:8020/accumulo", "2a", "default_tablet", "F0000070.rf");
+ test("hdfs://nn1:9000/accumulo/tables/5a/t-0005/C0009.rf", "hdfs://nn1:9000/accumulo", "5a",
+ "t-0005", "C0009.rf");
+ test(
+ "file:/home/dude/workspace/accumulo/test/target/mini-tests/org.apache.accumulo.test.VolumeIT_test/volumes/v1/tables/1/t-0000003/F0000006.rf",
+ "file:/home/dude/workspace/accumulo/test/target/mini-tests/org.apache.accumulo.test.VolumeIT_test/volumes/v1",
+ "1", "t-0000003", "F0000006.rf");
+ }
+
+ @Test
+ public void testBadPaths() {
+ try {
+ test("C0004.rf", "", "2a", "t-0003", "C0004.rf");
+ fail("Failed to throw error on bad path");
+ } catch (NullPointerException e) {}
+ // 2a< srv:dir
+ try {
+ test("dir", "", "2a", "", "");
+ fail("Failed to throw error on bad path");
+ } catch (NullPointerException e) {}
+ try {
+ test("hdfs://localhost:8020/accumulo/tablets/2a/default_tablet/F0000070.rf",
+ "hdfs://localhost:8020/accumulo", "2a", "default_tablet", "F0000070.rf");
+ fail("Failed to throw error on bad path");
+ } catch (IllegalArgumentException e) {}
+ try {
+ test("hdfs://localhost:8020/accumulo/2a/default_tablet/F0000070.rf",
+ " hdfs://localhost:8020/accumulo", "2a", "default_tablet", " F0000070.rf");
+ } catch (IllegalArgumentException e) {}
+ try {
+ test("/accumulo/tables/2a/default_tablet/F0000070.rf", "", "2a", "default_tablet",
+ "F0000070.rf");
+ } catch (IllegalArgumentException e) {}
+ try {
+ test("hdfs://localhost:8020/accumulo/tables/2a/F0000070.rf", "hdfs://localhost:8020/accumulo",
+ "2a", "", "F0000070.rf");
+ } catch (IllegalArgumentException e) {}
+ try {
+ test("hdfs://localhost:8020/accumulo/tables/F0000070.rf", "hdfs://localhost:8020/accumulo",
+ null, "", "F0000070.rf");
+ } catch (IllegalArgumentException e) {}
+ }
+
+ private final String id = "2a";
+ private final String dir = "t-0003";
+ private final String filename = "C0004.rf";
+
+ @Test
+ public void testFullPathWithVolume() {
+ String volume = "hdfs://1.2.3.4/accumulo";
+ String metadataEntry = volume + "/tables/" + id + "/" + dir + "/" + filename;
+ test(metadataEntry, volume, id, dir, filename);
+ }
+
+ @Test
+ public void testNormalizePath() {
+ String uglyVolume = "hdfs://nn.somewhere.com:86753/accumulo/blah/.././/bad/bad2/../.././/////";
+ String metadataEntry = uglyVolume + "/tables/" + id + "/" + dir + "/" + filename;
+ test(metadataEntry, "hdfs://nn.somewhere.com:86753/accumulo", id, dir, filename);
Review comment:
Would also be nice to verify that TabletFile.equals and TabletFile.hashCode work as expected for normalized paths.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services