You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "ctubbsii (via GitHub)" <gi...@apache.org> on 2023/08/29 02:28:01 UTC
[GitHub] [accumulo] ctubbsii commented on issue #3732: We really need to use formatters
ctubbsii commented on issue #3732:
URL: https://github.com/apache/accumulo/issues/3732#issuecomment-1696675223
First, I just want to mention I'm not sure this is the best venue for a discussion. This is probably better suited to the developer mailing list. But I'll respond here for now.
It's worth noting that this was something that we didn't quite finish cleaning up in 2.1 and 3.0 (my fault... I had this on my personal TODO list, but it got lost), so some stuff is still there and hasn't been removed. The scan interpreter was removed, but the formatter still partially remains as a per-table property. I would argue for completing that removal, rather than reversing course, though.
For background, there is quite a history with this. The earliest I could find relates to a proposal in 2013 to try to replace the shell with a suite of more narrowly tailored utilities [ACCUMULO-1045](https://issues.apache.org/jira/browse/ACCUMULO-1045) in the spirit of Unix's "Make each program do one thing well" philosophy, so that a bloated shell was not needed, and we could reduce a lot of complexity by empowering users to utilize tools in their OS's shell rather than reimplement those features ourselves in Accumulo's shell with a lot of maintenance overhead. Formatting the output using Linux tools (sed, awk, perl, cut, tr, head, tail, paste, etc.) was just one of the possible options. Now that we have modern Java, which can execute a .java source file with a main method without first compiling and packaging it, or that can use JShell, which comes loaded with a bunch of features itself for interactively scripting Java, and now that we have a simple to use AccumuloClient object
that can be initialized automatically using a client configuration file, it is very trivial to interact with Accumulo without using the provided shell. So, we are closer today than we ever have been to be able to get rid of the shell in favor of separate tools, though not quite in the way I proposed in ACCUMULO-1045. Now, most of the shell's interactivity features could probably be replaced by JShell, and the few remaining use cases can be easily scripted by users on-demand for their specific cases. So, in a way, I think we're even better situated to get rid of the Accumulo shell than what I proposed on that old JIRA.
We revisited the discussion when we started looking at the complexity and inconvenience of the ScanInterpreter in 2019 with #1138. It was there that we first actually proposed to remove the formatter and interpreter options, without a drastic change in the Accumulo shell. It finally got done over a year ago when #2806 was merged to address #2787, which itself was created because of the user confusion the current code caused (#2210).
While this change happened recently, it's been in the works for years, with on-and-off discussions about it. It basically boils down to this:
* The feature adds a lot of complexity, for a small convenience. There are now many much more simple ways for users to achieve the same thing. Even if JShell isn't used, it's trivial to write a small AccumuloClient that uses the same config file and scans a table and outputs lines in any format the user wishes.
* This feature tried to bridge the gap for users between the limited maintenance functionality that the shell is supposed to be for, and the way they actually think about their data. Unfortunately, it did so by creating confusion because Accumulo now has two views of the data... how it's actually stored, and the way users think about it; this creates as many problems as it solves, not the least of which was that it did this inconsistently between similar shell commands.
* It bloats the shell and makes users more dependent on Accumulo's shell to satisfy every little niche use case they might have, rather than being a general purpose tool for maintenance. We want to balance the convenience with having a general purpose maintenance tool with the shell having a limited and easy-to-maintain scope. We don't want to go too far and become the default means by which users scan their data, with a bloated and feature-full tool that is intended to handle nearly every use case out of an abundance of convenience (that's not sustainable).
Ultimately, I don't think we need this feature in Accumulo anymore to support the use case it was originally intended for. I think I could probably help you write a set of much more narrowly-focused set of tools to handle your use case. Before we try to reintroduce this feature, I'm happy to try to discuss your specific use cases, and to help your team consider some alternatives that I think might be better overall.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org