You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/09/18 15:14:43 UTC
[GitHub] [accumulo] cshannon opened a new issue, #2940: du command output format improvements
cshannon opened a new issue, #2940:
URL: https://github.com/apache/accumulo/issues/2940
This is a follow on to Issue #2820 and PR #2900 once merged. That PR improves the du command by changing the file size information source from being HDFS to the Accumulo metadata table which will improve performance with the trade off being that the size information will be estimated.
Further enhancements could also be done to the du command to provide more information or different output that could be of use. The current command only displays a limited amount of information (usage of a table or shared usage across tables for a pattern) but there is a lot of other useful information that could be displayed such as total usage by table, usage by volume, flags to indicate shared files, etc. The command could also be improved by adding different output formats depending on what the user needs such as a compact 1 line format per table that is easy to parse/grep, JSON output, pagination, a verbose mode for detailed shared usage information, etc.
One thing to note that if we do want to do this and enhance the output we will likely need to track more information during the compaction and return it. This would mean creating a new/updated thrift API call that returns more statistics when computing size information (assuming we still want to compute this on the server side but could also be done with a client scan).
#### Here are some examples of how things could look that came from the previous discussion on #2900:
#### Using a Pattern for multiple tables:
**Existing:**
```
root@uno t1> du -p t.*
310 [t1]
270 [t2, t3]
232 [t3]
```
**New:**
```
root@uno t1> dumeta -p t.* -v
t1 1 used total: 310 hdfs://localhost:8020/accumulo: 310 has_shared: false
t2 2 used total: 270 hdfs://localhost:8020/accumulo: 270 has_shared: false
t3 3 used total: 502 hdfs://localhost:8020/accumulo: 502 has_shared: true shared with: [2]
Shared usage between tables:
310 [t1]
270 [t2, t3]
232 [t3]
```
#### A single table
**Existing:**
```
root@uno t1> du t1
310 [t1]
```
**New:**
```
root@uno t1> dumeta t1
t1 1 used total: 310 hdfs://localhost:8020/accumulo: 310 has_shared: false
```
```
root@uno t1> dumeta t1 -v
t1 1 used total: 310 hdfs://localhost:8020/accumulo: 310 has_shared: false
Shared usage between tables:
310 [t1]
```
Note that in some cases my command can actually output more as it detects the shared table from the metadata even if only 1 table given. For example:
#### A shared table
**Existing:**
```
root@uno> du t3
502 [t3]
```
**New:**
```
root@uno> dumeta t3
t3 3 used total: 502 hdfs://localhost:8020/accumulo: 502 has_shared: true shared with: [2]
```
```
root@uno> dumeta t3 -v
t3 3 used total: 502 hdfs://localhost:8020/accumulo: 502 has_shared: true shared with: [2]
Shared usage between tables:
270 [t2, t3]
232 [t3]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #2940: du command output format improvements
Posted by GitBox <gi...@apache.org>.
EdColeman commented on issue #2940:
URL: https://github.com/apache/accumulo/issues/2940#issuecomment-1281245777
Closed by https://github.com/apache/accumulo/pull/2900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] EdColeman closed issue #2940: du command output format improvements
Posted by GitBox <gi...@apache.org>.
EdColeman closed issue #2940: du command output format improvements
URL: https://github.com/apache/accumulo/issues/2940
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org