You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "swamirishi (via GitHub)" <gi...@apache.org> on 2023/03/24 17:50:17 UTC
[GitHub] [ozone] swamirishi opened a new pull request, #4470: HDDS-8277. Parsing Bug in SSTDumptool
swamirishi opened a new pull request, #4470:
URL: https://github.com/apache/ozone/pull/4470
## What changes were proposed in this pull request?
Parsing SSTDumptool output fails when keys have single quotes & =<> characters in them since keys & value are concatenated together.
This patch looks into prepending the key length & value length in the dump tool output, which would concretely tell the lengths to parse them out.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8277
## How was this patch tested?
Manual testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4470: HDDS-8277. [Snapshot] Parsing Bug in SSTDumptool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4470:
URL: https://github.com/apache/ozone/pull/4470#discussion_r1173083494
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -43,18 +43,18 @@
private static final Logger LOG =
LoggerFactory.getLogger(ManagedSSTDumpIterator.class);
+ // Since we don't have any restriction on the key, we are prepending
+ // the length of the pattern in the sst dump tool output.
+ // The first token in the pattern is the key.
+ // The second tells the sequence number of the key.
+ // The third token gives the type of key in the sst file.
private static final String PATTERN_REGEX =
- "'([^=>]+)' seq:([0-9]+), type:([0-9]+) => ";
-
+ "'([\\s\\S]+)' seq:([0-9]+), type:([0-9]+)";
Review Comment:
```suggestion
"'([\\s\\S]+)' seq:([0-9]+), type:([0-9]+)";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde commented on a diff in pull request #4470: HDDS-8277. [snapshot] Parsing Bug in SSTDumptool
Posted by "prashantpogde (via GitHub)" <gi...@apache.org>.
prashantpogde commented on code in PR #4470:
URL: https://github.com/apache/ozone/pull/4470#discussion_r1149740209
##########
hadoop-hdds/rocks-native/src/main/patches/rocks-native.patch:
##########
@@ -1,20 +1,8 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
+commit 221a465ce6033c8b71503f156d9c8c9e011f79f3
Review Comment:
why did we remove the Apache license from here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #4470: HDDS-8277. [Snapshot] Parsing Bug in SSTDumptool
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #4470:
URL: https://github.com/apache/ozone/pull/4470#discussion_r1173122227
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -43,18 +43,18 @@
private static final Logger LOG =
LoggerFactory.getLogger(ManagedSSTDumpIterator.class);
+ // Since we don't have any restriction on the key, we are prepending
+ // the length of the pattern in the sst dump tool output.
+ // The first token in the pattern is the key.
+ // The second tells the sequence number of the key.
+ // The third token gives the type of key in the sst file.
private static final String PATTERN_REGEX =
- "'([^=>]+)' seq:([0-9]+), type:([0-9]+) => ";
-
+ "'([\\s\\S]+)' seq:([0-9]+), type:([0-9]+)";
Review Comment:
Need to fix other continuation indentations as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde commented on a diff in pull request #4470: HDDS-8277. [snapshot] Parsing Bug in SSTDumptool
Posted by "prashantpogde (via GitHub)" <gi...@apache.org>.
prashantpogde commented on code in PR #4470:
URL: https://github.com/apache/ozone/pull/4470#discussion_r1149736226
##########
hadoop-hdds/rocks-native/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpIterator.java:
##########
@@ -35,21 +36,14 @@
*/
public class ManagedSSTDumpIterator implements
Iterator<ManagedSSTDumpIterator.KeyValue>, AutoCloseable {
- private static final String SST_DUMP_TOOL_CLASS =
- "org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool";
private static final String PATTERN_REGEX =
- "'([^=>]+)' seq:([0-9]+), type:([0-9]+) => ";
-
+ "'([\\s\\S]+)' seq:([0-9]+), type:([0-9]+)";
Review Comment:
Can you put some comments here with some context/examples of what is relevant to us?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on a diff in pull request #4470: HDDS-8277. [snapshot] Parsing Bug in SSTDumptool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #4470:
URL: https://github.com/apache/ozone/pull/4470#discussion_r1160237540
##########
hadoop-hdds/rocks-native/src/main/patches/rocks-native.patch:
##########
@@ -1,20 +1,8 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
+commit 221a465ce6033c8b71503f156d9c8c9e011f79f3
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4470: HDDS-8277. Parsing Bug in SSTDumptool
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4470:
URL: https://github.com/apache/ozone/pull/4470#issuecomment-1483210892
Please fix rat failure:
https://github.com/swamirishi/ozone/actions/runs/4513912390/jobs/7949343747#step:6:814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swamirishi commented on pull request #4470: HDDS-8277. [Snapshot] Parsing Bug in SSTDumptool
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #4470:
URL: https://github.com/apache/ozone/pull/4470#issuecomment-1513291633
@prashantpogde Can this PR be merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde merged pull request #4470: HDDS-8277. [Snapshot] Parsing Bug in SSTDumptool
Posted by "prashantpogde (via GitHub)" <gi...@apache.org>.
prashantpogde merged PR #4470:
URL: https://github.com/apache/ozone/pull/4470
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org