You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Rick Hillegas (JIRA)" <ji...@apache.org> on 2011/04/15 20:01:06 UTC
[jira] [Created] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Create tools for browsing the files in the database log directory.
------------------------------------------------------------------
Key: DERBY-5195
URL: https://issues.apache.org/jira/browse/DERBY-5195
Project: Derby
Issue Type: Task
Affects Versions: 10.9.0.0
Reporter: Rick Hillegas
It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rick Hillegas updated DERBY-5195:
---------------------------------
Attachment: LogFileReader.java
Attaching a new version of LogFileReader. This version adds support for pretty-printing log files in encrypted databases:
Usage:
java LogFileReader $logFileName [ -v ] [ -p $P ] [ -n $N ] [ -e $bootPassword $serviceProperties ]
-v Verbose. Deserialize the logged operations. If you do not set this flag, the tool just decodes the wrapper headers.
-p Starting position. $P is a positive number, the offset of the first log entry to read. This causes the tool to skip reading the file header as well.
-n Number of records to read. $N is a non-negative number. If you do not specify this flag, the tool prints all subsequent log entries.
-e If the database is encrypted, you must supply the boot password and the location of service.properties.
The following example decrypts and deserializes an entire log file, dumping the result into an xml file for inspection:
java LogFileReader wombat/log/log1.dat -v -e Wednesday wombat/service.properties > z.xml
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java, ControlFileReader.java, LogFileReader.java, LogFileReader.java, LogFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rick Hillegas updated DERBY-5195:
---------------------------------
Attachment: LogFileReader.java
Attaching new version of LogFileReader.java. This version tries to do a better job of reporting more information before exiting after tripping across a corrupt record.
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java, ControlFileReader.java, LogFileReader.java, LogFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rick Hillegas updated DERBY-5195:
---------------------------------
Attachment: ControlFileReader.java
Attaching ControlFileReader.java. This tool reads the control file in the log directory and pretty-prints it as an xml file. I have run this tool against log.ctrl files created by 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, and trunk (10.9) databases and the tool seems to handle them ok. Here's what running the tool looks like:
java ControlFileReader db/log/log.ctrl
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<controlFile>
<formatableID>128</formatableID>
<obsoleteVersion>9</obsoleteVersion>
<logCounter>
<logFileNumber>1</logFileNumber>
<position>11369</position>
</logCounter>
<majorVersion>10</majorVersion>
<minorVersion>9</minorVersion>
<subversionRevision>1092193</subversionRevision>
<flags flags="1">
<flag>IS_BETA_FLAG</flag>
</flags>
</controlFile>
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Mike Matrigali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030175#comment-13030175 ]
Mike Matrigali commented on DERBY-5195:
---------------------------------------
o personally I would rather not see these types of tools added as supported to the DERBY project, but rather either as unsupported
SANE tools or as tools on top of derby in another project. I don't think the production derby.jar should be weighed down with debugging code/tools.
If we decide to start "supporting" more
tools i think they should be checked into a different part of the codeline and a different jar file. I do think it would be nice if there
was a place to check in unsupported tools on top of DERBY, and we have discussed it in the past but nothing came of it. A number
of other things come to mind like user definined functions and datatypes. I think there have been some useful VTI's that would
also fit into this.
o I scanned the DataReader file and it looked like it did not depend on store at all, so would also benefit from not being in the
store file structure so that it keeps that independence.
o I also worry if we productize these then they could be used in unintended ways and may cause problems by their use. The intent
is to use single user on a copy of the database, but what happens if people start running these against a running database either
by choice or accidently. A lot of the existing corruptions of DERBY have come from users manipulating the log and datafiles by hand. I
would rather not encourage more to do so.
o These tools basically let anyone scan a table (sort of), how does that fit in with SQL security?
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java, ControlFileReader.java, LogFileReader.java, LogFileReader.java, LogFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029962#comment-13029962 ]
Kathey Marsden commented on DERBY-5195:
---------------------------------------
Thank you Rick for making this tool. It will be extremely helpful to people supporting Derby.
It seems to me that this should be checked in somewhere and if deployed with the product jars even better, but I am not sure the best place. Do you have thoughts on this?
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java, ControlFileReader.java, LogFileReader.java, LogFileReader.java, LogFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022280#comment-13022280 ]
Rick Hillegas commented on DERBY-5195:
--------------------------------------
As a result of writing LogFileReader, I have verified that the format of the log file is correctly documented here: http://db.apache.org/derby/papers/logformats.html
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java, ControlFileReader.java, LogFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rick Hillegas updated DERBY-5195:
---------------------------------
Attachment: ControlFileReader.java
Attaching a second rev of ControlFileReader, which includes the checksum field.
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java, ControlFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Mike Matrigali (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Matrigali updated DERBY-5195:
----------------------------------
Component/s: Tools
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Components: Tools
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java, ControlFileReader.java, LogFileReader.java, LogFileReader.java, LogFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030774#comment-13030774 ]
Rick Hillegas commented on DERBY-5195:
--------------------------------------
Thanks for the comments, Mike. Some more discussion below...
MM> o personally I would rather not see these types of tools added as supported to the DERBY project, but rather either as unsupported
SANE tools or as tools on top of derby in another project. I don't think the production derby.jar should be weighed down with debugging code/tools.
If we decide to start "supporting" more
tools i think they should be checked into a different part of the codeline and a different jar file. I do think it would be nice if there
was a place to check in unsupported tools on top of DERBY, and we have discussed it in the past but nothing came of it. A number
of other things come to mind like user definined functions and datatypes. I think there have been some useful VTI's that would
also fit into this.
It might make sense to distinguish between supported and unsupported tools. I have attached a number of tools to JIRA issues. Technically I don't consider myself responsible for fixing them when they don't work as advertised. Practically speaking, I have been pretty responsive about fixing these tools when complaints come in. It would be pretty funny if the distinction between unsupported and supported turned out to be this: for an unsupported tool, a Derby developer gives you a fix right away, whereas for a supported tool, you won't get a fix until the next Derby release.
Another way to distinguish tools would be by who the target users are. The programs in derbytools.jar seem to be aimed at the developers who write Derby-powered apps, the dbas who administer those apps in production, and the tech support engineers who field bugs logged against those apps. In contrast, I suspect that ControlFileReader, LogFileReader, and DataFileReader will be used mostly by derby-dev developers, that is, us. It's possible that these tools might be used by tech support engineers who support specific Derby-powered apps--but I think that is highly unlikely. I don't think those engineers want to become experts in Derby's internal formats. To me, these new tools seem most useful when the bug ends up in Derby's JIRA tracker.
Don't know if that helps push the conversation forward, but I thought it was worth saying.
MM> o I scanned the DataReader file and it looked like it did not depend on store at all, so would also benefit from not being in the
store file structure so that it keeps that independence.
LogFileReader imports org.apache.derby.iapi.store.raw.Loggable in order to get its hands on some constants. It would be easy to break that dependency. Both DataFileReader and LogFileReader depend on other machinery in the engine, chiefly code under org.apache.derby.iapi.services.
MM> o I also worry if we productize these then they could be used in unintended ways and may cause problems by their use. The intent
is to use single user on a copy of the database, but what happens if people start running these against a running database either
by choice or accidently. A lot of the existing corruptions of DERBY have come from users manipulating the log and datafiles by hand. I
would rather not encourage more to do so.
Don't know how these interactions would play out. The new tools merely read the files in log and seg0. The tools never change the files though. Don't know if merely reading these files would trip up another Derby engine.
MM> o These tools basically let anyone scan a table (sort of), how does that fit in with SQL security?
Right. DataFileReader lets you print out a table if you have access to the database directory and know both the table's conglomerate number and its row signature. A moderately competent engineer could write this tool in less than a week provided that they knew about the data formats described here: http://db.apache.org/derby/papers/pageformats.html So, the tool costs maybe a couple thousand dollars to write. Amortized over just a couple criminal customers, the cost per crime could be even lower. Note also that you can view a surprising amount of a Derby database by just cat'ing the files in seg0. I would agree that the tools slightly increase the vulnerability of low value data.
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java, ControlFileReader.java, LogFileReader.java, LogFileReader.java, LogFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rick Hillegas updated DERBY-5195:
---------------------------------
Attachment: LogFileReader.java
Attaching LogFileReader.java, a program for pretty-printing the contents of Derby log files.
I have successfully run this tool against log files created by 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, and trunk codelines.
Here is the usage summary:
java LogFileReader
Usage:
java LogFileReader $logFileName [ -v ] [ -p $P ] [ -n $N ]
-v Verbose. Deserialize the logged operations. If you do not set this flag, the tool just decodes the wrapper headers.
-p Starting position. $P is a positive number, the offset of the first log entry to read. This causes the tool to skip reading the file header as well.
-n Number of records to read. $N is a non-negative number. If you do not specify this flag, the tool prints all subsequent log entries.
Here are some examples.
------------------------------------------------------------
1) Decode an entire log file, putting the resulting xml in the file z.xml. You can then view that file using a browser like Firefox, which lets you collapse and expand the elements. Because the -v switch is specified, the contents of the logged operations are deserialized and <details> elements are populated with the toString() results:
runjava LogFileReader db/log/log2.dat -v > z.xml
------------------------------------------------------------
2) Pretty-print a log file header:
java LogFileReader db/log/log2.dat -n 0
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<logFile>
<formatableID>128</formatableID>
<obsoleteVersion>9</obsoleteVersion>
<logFileNumber>2</logFileNumber>
<previousLogRecordEndInstant>
<logFileNumber>1</logFileNumber>
<position>11510</position>
</previousLogRecordEndInstant>
<logRecords>
</logRecords>
<recordCount>0</recordCount>
</logFile>
------------------------------------------------------------
3) Count the number of entries in a log file:
java LogFileReader db/log/log2.dat | grep recordCount
<recordCount>156</recordCount>
------------------------------------------------------------
4) Decode 3 log entries, starting at a given record offset:
java LogFileReader db/log/log2.dat -v -p 29363 -n 3
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<logFile>
<logRecords>
<wrapper>
<forwardLength>1818</forwardLength>
<logInstant>
<logFileNumber>2</logFileNumber>
<position>29363</position>
</logInstant>
<logRecord>
<formatableID>129</formatableID>
<groups hexvalue="180">
<flag>XA_NEEDLOCK</flag>
<flag>RAWSTORE</flag>
</groups>
<transactionID value="192"/>
<operation type="org.apache.derby.impl.store.raw.data.PurgeOperation">
<details>Page Operation: Page(2,Container(0, 81)) pageVersion 75 : Purge : 36 slots starting at 37 (recordId=43) (recordId=44) (recordId=45) (recordId=46) (recordId=47) (recordId=48) (recordId=49) (recordId=50) (recordId=51) (recordId=52) (recordId=53) (recordId=54) (recordId=55) (recordId=56) (recordId=57) (recordId=58) (recordId=59) (recordId=60) (recordId=61) (recordId=62) (recordId=63) (recordId=64) (recordId=65) (recordId=66) (recordId=67) (recordId=68) (recordId=69) (recordId=70) (recordId=71) (recordId=72) (recordId=73) (recordId=74) (recordId=75) (recordId=76) (recordId=77) (recordId=78)</details>
</operation>
</logRecord>
<backwardLength>1818</backwardLength>
</wrapper>
<wrapper>
<forwardLength>90</forwardLength>
<logInstant>
<logFileNumber>2</logFileNumber>
<position>31197</position>
</logInstant>
<logRecord>
<formatableID>129</formatableID>
<groups hexvalue="180">
<flag>XA_NEEDLOCK</flag>
<flag>RAWSTORE</flag>
</groups>
<transactionID value="192"/>
<operation type="org.apache.derby.impl.store.raw.data.InsertOperation">
<details>Page Operation: Page(1,Container(0, 81)) pageVersion 147 : Insert : Slot=1 recordId=79</details>
</operation>
</logRecord>
<backwardLength>90</backwardLength>
</wrapper>
<wrapper>
<forwardLength>17</forwardLength>
<logInstant>
<logFileNumber>2</logFileNumber>
<position>31303</position>
</logInstant>
<logRecord>
<formatableID>129</formatableID>
<groups hexvalue="112">
<flag>LAST</flag>
<flag>COMMIT</flag>
<flag>RAWSTORE</flag>
</groups>
<transactionID value="192"/>
<operation type="org.apache.derby.impl.store.raw.xact.EndXact">
<details>EndXact null Committed : transactionStatus = Committed</details>
</operation>
</logRecord>
<backwardLength>17</backwardLength>
</wrapper>
</logRecords>
<recordCount>3</recordCount>
</logFile>
------------------------------------------------------------
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java, ControlFileReader.java, LogFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5195) Create tools for browsing the files
in the database log directory.
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029981#comment-13029981 ]
Rick Hillegas commented on DERBY-5195:
--------------------------------------
Hi Kathey,
I agree that it would help tech support if we checked in DataFileReader, ControlFileReader, and LogFileReader. Together they weigh a little under 60K so I don't think we would significantly bloat up the engine jar if we checked them into an engine directory. Maybe org.apache.derby.impl.store.tools? Mike may have some better theory about where to put these tools.
Productizing these tools would be a little chunk of work:
1) Refactor the tools so that they share helper code which is currently duplicated across them.
2) Wire them into the list of extra classes so that the jar builder puts them into derby.jar.
3) Write regression tests to verify their behavior.
4) Document them in the Tools Guide.
Thanks,
-Rick
> Create tools for browsing the files in the database log directory.
> ------------------------------------------------------------------
>
> Key: DERBY-5195
> URL: https://issues.apache.org/jira/browse/DERBY-5195
> Project: Derby
> Issue Type: Task
> Affects Versions: 10.9.0.0
> Reporter: Rick Hillegas
> Attachments: ControlFileReader.java, ControlFileReader.java, LogFileReader.java, LogFileReader.java, LogFileReader.java
>
>
> It would be nice to have some tools to examine the contents of the log directory. The tools should be able to read the files in that directory without disturbing them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira