You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Michele Mazzucco <Mi...@ncl.ac.uk> on 2006/09/09 21:18:37 UTC
[io] new ThresholdingOutputStream implementation
Hi all,
I've extended the ThresholdingOutputStream class with a new class which
behaves different from DeferredFileOutputStream:
- when the stream is closed, the content stored in memory is *always*
flushed to disk (in DeferredFileOutputStream, instead, if the treshold
is not reached data is lost)
- DeferredFileOutputStream maintains data in memory only until the
treshold value has been reached, then it immediately writes every byte
to disk. Mine implementation, instead, caches treshold bytes in memory,
and every time that value is reached (that is, treshold, 2 * threshold,
etc), it flushes data to disk. In other words it acts as a cache.
Please find attached the class together with the unit test.
Best regards,
Michele
Re: [io] new ThresholdingOutputStream implementation
Posted by Michele Mazzucco <Mi...@ncl.ac.uk>.
JIRA IO-92 created.
Michele
Stephen Colebourne wrote:
> Sounds interesting. Can you raise a JIRA call for this (enhancement) and
> attach the changes to it? That way we won't lose it. Thanks
> Stephen
>
> Michele Mazzucco wrote:
>> Hi all,
>>
>> I've extended the ThresholdingOutputStream class with a new class which
>> behaves different from DeferredFileOutputStream:
>> - when the stream is closed, the content stored in memory is *always*
>> flushed to disk (in DeferredFileOutputStream, instead, if the treshold
>> is not reached data is lost)
>> - DeferredFileOutputStream maintains data in memory only until the
>> treshold value has been reached, then it immediately writes every byte
>> to disk. Mine implementation, instead, caches treshold bytes in memory,
>> and every time that value is reached (that is, treshold, 2 * threshold,
>> etc), it flushes data to disk. In other words it acts as a cache.
>>
>> Please find attached the class together with the unit test.
>>
>>
>>
>> Best regards,
>> Michele
>>
>>
>> ------------------------------------------------------------------------
>>
>> /*
>> * Licensed to the Apache Software Foundation (ASF) under one or more
>> * contributor license agreements. See the NOTICE file distributed with
>> * this work for additional information regarding copyright ownership.
>> * The ASF licenses this file to You under the Apache License, Version
>> 2.0
>> * (the "License"); you may not use this file except in compliance with
>> * the License. You may obtain a copy of the License at
>> * * http://www.apache.org/licenses/LICENSE-2.0
>> * * Unless required by applicable law or agreed to in writing, software
>> * distributed under the License is distributed on an "AS IS" BASIS,
>> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>> implied.
>> * See the License for the specific language governing permissions and
>> * limitations under the License.
>> */
>>
>> package org.apache.commons.io.output;
>>
>> import java.io.DataInputStream;
>> import java.io.File;
>> import java.io.FileInputStream;
>> import java.io.FileNotFoundException;
>> import java.io.IOException;
>> import java.util.Arrays;
>>
>> import junit.framework.TestCase;
>>
>> /**
>> * JUnit test case for {@link DeferredPeriodicOutputStream}.
>> * * @author <a href="mailto:Michele.Mazzucco@ncl.ac.uk">Michele
>> Mazzucco</a>
>> *
>> */
>> public class DeferredPeriodicOutputStreamTest extends TestCase {
>>
>>
>> /**
>> * The test data as a string (which is the simplest form).
>> */
>> private String testString = "0123456789";
>>
>> /**
>> * The test data as a byte array, derived from the string.
>> */
>> private byte[] testBytes = testString.getBytes();
>>
>>
>> /**
>> * Standard JUnit test case constructor.
>> * * @param name
>> * The name of the test case.
>> */
>> public DeferredPeriodicOutputStreamTest(String name) {
>> super(name);
>> }
>>
>>
>> /**
>> * Tests the case where the amount of data falls below the
>> threshold, and is
>> * therefore confined to memory.
>> */
>> public void testBelowThreshold() {
>> File testFile = getTestFile("testBelowTreshold.dat");
>>
>> DeferredPeriodicOutputStream dpos =
>> new DeferredPeriodicOutputStream(testBytes.length +
>> 42, testFile);
>> try {
>> dpos.write(testBytes, 0, testBytes.length);
>> dpos.close();
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> byte[] resultBytes = dpos.getData();
>> assertEquals(0, resultBytes.length);
>> verifyResultFile(testFile);
>> }
>>
>> /**
>> * Tests the case where the amount of data is exactly the same as the
>> * threshold. The behavior should be the same as that for the
>> amount of
>> * data being below (i.e. not exceeding) the threshold.
>> */
>> public void testAtThreshold() {
>> File testFile = getTestFile("testAtThreshold.dat");
>>
>> DeferredPeriodicOutputStream dpos =
>> new DeferredPeriodicOutputStream(testBytes.length,
>> testFile);
>> try
>> {
>> dpos.write(testBytes, 0, testBytes.length);
>> dpos.close();
>> }
>> catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> byte[] resultBytes = dpos.getData();
>> assertEquals(0, resultBytes.length);
>> verifyResultFile(testFile);
>> }
>> /**
>> * Tests the case where the amount of data exceeds the threshold,
>> and is
>> * therefore written to disk. The actual data written to disk is
>> verified,
>> * as is the file itself.
>> */
>> public void testAboveThreshold() {
>> File testFile = getTestFile("testAboveThreshold.dat");
>> DeferredFileOutputStream dfos =
>> new DeferredFileOutputStream(testBytes.length - 5,
>> testFile);
>> try
>> {
>> dfos.write(testBytes, 0, testBytes.length);
>> dfos.close();
>> }
>> catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>> assertFalse(dfos.isInMemory());
>> assertNull(dfos.getData());
>>
>> verifyResultFile(testFile);
>> }
>> /**
>> * Verifies that the specified file contains the same data as the
>> original
>> * test data.
>> *
>> * @param testFile The file containing the test output.
>> */
>> private void verifyResultFile(File testFile) {
>> try
>> {
>> FileInputStream fis = new FileInputStream(testFile);
>> assertTrue(fis.available() == testBytes.length);
>>
>> byte[] resultBytes = new byte[testBytes.length];
>> assertTrue(fis.read(resultBytes) == testBytes.length);
>>
>> assertTrue(Arrays.equals(resultBytes, testBytes));
>> assertTrue(fis.read(resultBytes) == -1);
>>
>> try
>> {
>> fis.close();
>> }
>> catch (IOException e) {
>> // Ignore an exception on close
>> }
>> }
>> catch (FileNotFoundException e) {
>> fail("Unexpected FileNotFoundException");
>> }
>> catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>> }
>>
>> public void testCheckThreshold() {
>> File testFile = getTestFile("testCheckThreshold.dat");
>>
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(8, testFile);
>>
>> try {
>> dpos.write(testBytes);
>> dpos.close();
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>> assertTrue(dpos.isThresholdExceeded());
>> }
>>
>> private static File getTestFile(String path) {
>> File testFile = new File(path);
>> testFile.deleteOnExit();
>> testFile.delete();
>>
>> return testFile;
>> }
>>
>> public void testGetStream() {
>> File testFile = getTestFile("testGetStream.dat");
>>
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(12, testFile);
>> try {
>> dpos.write(testBytes);
>> byte[] memory = dpos.getData();
>> assertEquals(memory.length, 10);
>> assertTrue(Arrays.equals(testBytes, memory));
>>
>> dpos.close();
>> memory = dpos.getData();
>> assertEquals(memory.length, 0);
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>> }
>>
>> /**
>> * Tests the write of a int value.
>> */
>> public void testWriteInt() {
>> File testFile = getTestFile("testWriteInt.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(5, testFile);
>>
>> try {
>> dpos.writeInt(5);
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> int read = getInt(dpos.getData());
>> assertTrue(read == 5);
>> }
>>
>> /**
>> * Tests the write of a float value.
>> */
>> public void testWriteFloat() {
>> File testFile = getTestFile("testWriteFloat.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(9, testFile);
>>
>> try {
>> dpos.writeFloat(15.65f);
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> float read = getFloat(dpos.getData());
>> assertTrue(read == 15.65f);
>> }
>>
>> /**
>> * Tests the write of a long value.
>> */
>> public void testWriteLong() {
>> File testFile = getTestFile("testWriteLong.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(9, testFile);
>>
>> long value = System.currentTimeMillis();
>> try {
>> dpos.writeLong(value);
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> float read = getLong(dpos.getData());
>> assertTrue(read == value);
>> }
>>
>> /**
>> * Tests the write of a double value.
>> */
>> public void testWriteDouble() {
>> File testFile = getTestFile("testWriteDouble.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(9, testFile);
>>
>> try {
>> dpos.writeDouble(Math.PI);
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> double read = getDouble(dpos.getData());
>> assertTrue(read == Math.PI);
>> }
>>
>>
>> /**
>> * Tests the write of a char value.
>> */
>> public void testWriteChar() {
>> File testFile = getTestFile("testWriteChar.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(3, testFile);
>>
>> try {
>> dpos.writeChar('b');
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> char c = getChar(dpos.getData());
>> assertTrue(c == 'b');
>> }
>>
>> /**
>> * Tests the write of a boolean value.
>> */
>> public void testWriteBoolean() {
>> File testFile = getTestFile("testWriteBoolean.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(2, testFile);
>>
>> try {
>> dpos.writeBoolean(true);
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> boolean read = (dpos.getData()[0] != 0);
>> assertTrue(read == true);
>> }
>>
>>
>> /**
>> * Tests the write of a byte value.
>> */
>> public void testWriteByte() {
>> File testFile = getTestFile("testWriteByte.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(2, testFile);
>>
>> try {
>> dpos.writeByte(3);
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> byte read = (byte) (dpos.getData()[0]);
>> assertTrue(read == 3);
>> }
>>
>> public void testWriteBytes() {
>> File testFile = getTestFile("testWriteBytes.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(2, testFile);
>>
>> try {
>> dpos.writeBytes(testString);
>> dpos.close();
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> verifyResultFile(testFile);
>> }
>>
>> public void testWriteChars() {
>> File testFile = getTestFile("testWriteChars.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(21, testFile);
>>
>> try {
>> dpos.writeChars(testString);
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> byte[] read = dpos.getData();
>> char[] c = new char[10];
>> byte[] b = new byte[2];
>> int j = 0;
>> for (int i = 0; i < read.length; i = (i + 2)) {
>> System.arraycopy(read, i, b, 0, 2);
>> c[j] = getChar(b);
>> j++;
>> }
>>
>> assertEquals(testString, new String(c));
>> }
>>
>> /**
>> * Tests the write of a short value.
>> */
>> public void testWriteShort() {
>> File testFile = getTestFile("testWriteShort.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(3, testFile);
>>
>> try {
>> dpos.writeShort(3);
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> short read = (short) getChar(dpos.getData());
>> assertTrue(read == 3);
>> }
>>
>> /**
>> * Tests the write of an UTF string.
>> */
>> public void testWriteUTF() {
>> File testFile = getTestFile("testWriteUTF.dat");
>> DeferredPeriodicOutputStream dpos = new
>> DeferredPeriodicOutputStream(3, testFile);
>>
>> try {
>> dpos.writeUTF(testString);
>> dpos.close();
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> }
>>
>> try {
>> DataInputStream dis = new DataInputStream(new
>> FileInputStream(testFile));
>> String s = dis.readUTF();
>>
>> assertEquals(testString, s);
>>
>> try {
>> dis.close();
>> } catch (IOException e) {
>> // Ignore this
>> }
>> } catch (FileNotFoundException e) {
>> fail("Unexpected FileNotFoundException");
>> } catch (IOException e) {
>> fail("Unexpected IOException");
>> } }
>>
>>
>>
>> //===================================================================//
>>
>> // Static methods used to convert array of bytes to primitive values
>> //===================================================================//
>>
>>
>> /**
>> * Converts the specified array of <code>bytes</code> to a
>> <code>int</code>.
>> * * @param b An array of <code>bytes</code>.
>> * @return The equivalent <code>int</code> value.
>> */
>> public static int getInt(byte[] b) {
>> assert b.length == 4: "Invalid number of bytes for integer
>> conversion";
>> return ((b[0] << 24) & 0xFF000000) + ((b[1] << 16) &
>> 0x00FF0000) +
>> ((b[2] << 8) & 0x0000FF00) + ((b[3] << 0) & 0x000000FF);
>> }
>>
>> /**
>> * Converts the specified array of <code>bytes</code> to a
>> <code>float</code>.
>> * * @param b An array of <code>bytes</code>.
>> * @return The equivalent <code>float</code> value.
>> */
>> public static float getFloat(byte[] b) {
>> assert b.length == 4: "Invalid number of bytes for float
>> conversion";
>> return Float.intBitsToFloat(getInt(b));
>> }
>>
>> /**
>> * Converts the specified array of <code>bytes</code> to a
>> <code>long</code>.
>> * * @param b An array of <code>bytes</code>.
>> * @return The equivalent <code>long</code> value.
>> */
>> public static long getLong(byte[] b) {
>> assert b.length == 8: "Invalid number of bytes for long
>> conversion";
>> int high = getInt(new byte[] { b[0], b[1], b[2], b[3] });
>> int low = getInt(new byte[] { b[4], b[5], b[6], b[7] });
>> long value = ((long)(high) << 32) + (low & 0xFFFFFFFFL);
>> return value;
>> }
>>
>> /**
>> * Converts the specified array of <code>bytes</code> to a
>> <code>double</code>.
>> * * @param b An array of <code>bytes</code>.
>> * @return The equivalent <code>double</code> value.
>> */
>> public static double getDouble(byte[] b) {
>> assert b.length == 8: "Invalid number of bytes for double
>> conversion";
>> return Double.longBitsToDouble(getLong(b));
>> }
>>
>> /**
>> * Converts the specified array of <code>bytes</code> to a
>> <code>char</code>.
>> * * @param b An array of <code>bytes</code>.
>> * @return The equivalent <code>char</code> value.
>> */
>> public static char getChar(byte[] b) {
>> assert b.length == 2: "Invalid number of bytes for char
>> conversion";
>> return (char)(((b[0] << 8) & 0x0000FF00) + ((b[1] << 0) &
>> 0x000000FF));
>> }
>> }
>>
>>
>> ------------------------------------------------------------------------
>>
>> /*
>> * Licensed to the Apache Software Foundation (ASF) under one or more
>> * contributor license agreements. See the NOTICE file distributed with
>> * this work for additional information regarding copyright ownership.
>> * The ASF licenses this file to You under the Apache License, Version
>> 2.0
>> * (the "License"); you may not use this file except in compliance with
>> * the License. You may obtain a copy of the License at
>> * * http://www.apache.org/licenses/LICENSE-2.0
>> * * Unless required by applicable law or agreed to in writing, software
>> * distributed under the License is distributed on an "AS IS" BASIS,
>> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>> implied.
>> * See the License for the specific language governing permissions and
>> * limitations under the License.
>> */
>>
>> package org.apache.commons.io.output;
>>
>> import java.io.DataOutput;
>> import java.io.File;
>> import java.io.FileOutputStream;
>> import java.io.IOException;
>> import java.io.OutputStream;
>> import java.io.UTFDataFormatException;
>>
>>
>> /**
>> * The <code>DeferredPeriodicOutputStream</code> is an output stream
>> which
>> * retains data in memory until a specified treshold is reached,
>> commit it
>> * to disk and reset the threshold value. The result is that data are
>> written
>> * to disk on a periodic basis, that is, every <i>treshold</i> bytes.
>> In other
>> * words, this output stream acts as a cache.
>> * <p>
>> * Opposite to {@link DeferredFileOutputStream}, if the stream is
>> * closed before the threshold is reached, the data are written to
>> * disk.
>> * * @author <a href="mailto:Michele.Mazzucco@ncl.ac.uk">Michele
>> Mazzucco</a>
>> * @version $Version$
>> * @see org.apache.commons.io.output.DeferredFileOutputStream
>> */
>> public class DeferredPeriodicOutputStream extends
>> ThresholdingOutputStream implements DataOutput {
>>
>>
>>
>> //=======================================================================//
>>
>> // Instance fields
>>
>> //=======================================================================//
>>
>>
>>
>> /**
>> * The output stream to which data will be written before every
>> theshold
>> * is reached.
>> */
>> private ByteArrayOutputStream memoryOutputStream;
>>
>> /**
>> * The output stream to which data will be written after every
>> theshold
>> * is reached.
>> */
>> private FileOutputStream diskOutputStream;
>>
>> /**
>> * The file to which output will be directed every time the
>> threshold is * exceeded.
>> */
>> private File outputFile;
>>
>>
>>
>>
>> //=======================================================================//
>>
>> // Constructor
>>
>> //=======================================================================//
>>
>>
>>
>> /**
>> * Creates a new <tt>DeferredPeriodicOutputStream</tt> object.
>> *
>> * @param threshold The number of bytes to keep in memory.
>> * @param file The file to use every time the threshold is
>> reached.
>> */
>> public DeferredPeriodicOutputStream(int threshold, File file) {
>> super(threshold);
>> this.outputFile = file;
>> this.memoryOutputStream = new ByteArrayOutputStream(threshold);
>> }
>>
>> /**
>> * Creates a new <tt>DeferredPeriodicOutputStream</tt> object.
>> *
>> * @param threshold The number of bytes to keep in memory.
>> * @param file The path to the file to use every time the
>> threshold * is reached.
>> */
>> public DeferredPeriodicOutputStream(int treshold, String path) {
>> this(treshold, new File(path));
>> }
>>
>>
>>
>> //=======================================================================//
>>
>> // Methods from ThresholdingOutputStream
>>
>> //=======================================================================//
>>
>>
>>
>> /**
>> * Gets the memory output stream (<i>always</i>).
>> * * @return The memory ouput stream.
>> */
>> @Override
>> protected OutputStream getStream() {
>> return memoryOutputStream;
>> }
>>
>> @Override
>> protected synchronized void thresholdReached() throws IOException {
>> if (diskOutputStream == null) {
>> diskOutputStream = new FileOutputStream(outputFile);
>> }
>> memoryOutputStream.writeTo(diskOutputStream);
>> diskOutputStream.flush();
>> memoryOutputStream.reset();
>> }
>>
>>
>> @Override
>> protected synchronized void checkThreshold(int count) throws
>> IOException {
>> if ((memoryOutputStream.size() + count) > super.getThreshold()) {
>> this.thresholdReached();
>> } }
>>
>> /**
>> * Closes this output stream flushing all cached data to disk.
>> */
>> @Override
>> public void close() throws IOException {
>> super.close();
>> if (diskOutputStream == null) {
>> if (memoryOutputStream.size() > 0) {
>> diskOutputStream = new FileOutputStream(outputFile);
>> } else {
>> return;
>> }
>> }
>>
>> memoryOutputStream.writeTo(diskOutputStream);
>> flush();
>> memoryOutputStream.reset();
>> memoryOutputStream.close();
>>
>> if (diskOutputStream != null) {
>> diskOutputStream.close();
>> }
>> }
>>
>>
>> /**
>> * Flushes this output stream and forces any buffered output bytes
>> to be
>> * written out.
>> *
>> * @exception IOException if an error occurs.
>> */
>> @Override
>> public void flush() throws IOException {
>> memoryOutputStream.flush();
>>
>> if (diskOutputStream != null)
>> diskOutputStream.flush();
>> }
>>
>> //
>> ======================================================================//
>> // Public methods
>>
>> //=======================================================================//
>>
>>
>>
>>
>> /**
>> * Returns an array of bytes containing between <code>0</code> and
>> * {@link ThresholdingOutputStream#threshold} bytes. The array
>> corresponds
>> * to the data cached in memory.
>> * * @return The data for this output stream which are
>> maintained in memory.
>> */
>> public byte[] getData() {
>> return memoryOutputStream.toByteArray();
>> }
>>
>>
>>
>> //=======================================================================//
>>
>> // Methods from java.io.DataOuput
>>
>> //=======================================================================//
>>
>>
>>
>> /**
>> * Writes an <code>int</code> to the underlying output stream as four
>> * bytes, high byte first. If no exception is thrown, the counter
>> * <code>written</code> is incremented by <code>4</code>.
>> *
>> * @param value an <code>int</code> to be written.
>> * @exception IOException if an I/O error occurs.
>> */
>> public void writeInt(int value) throws IOException {
>> super.write(getIntBytes(value));
>> }
>>
>> /**
>> * Converts the float argument to an <code>int</code> using the
>> * <code>floatToIntBits</code> method in class <code>Float</code>,
>> * and then writes that <code>int</code> value to the underlying
>> * output stream as a 4-byte quantity, high byte first. If no
>> * exception is thrown, the counter <code>written</code> is *
>> incremented by <code>4</code>.
>> *
>> * @param value a <code>float</code> value to be written.
>> * @exception IOException if an I/O error occurs.
>> * @see java.lang.Float#floatToIntBits(float)
>> */
>> public void writeFloat(float value) throws IOException {
>> writeInt(Float.floatToIntBits(value));
>> }
>>
>> /**
>> * Writes a <code>long</code> to the underlying output stream as
>> eight
>> * bytes, high byte first. In no exception is thrown, the counter
>> * <code>written</code> is incremented by <code>8</code>.
>> *
>> * @param value a <code>long</code> to be written.
>> * @exception IOException if an I/O error occurs.
>> * @see java.io.FilterOutputStream#out
>> */
>> public void writeLong(long value) throws IOException {
>> super.write(getLongBytes(value));
>> }
>>
>> /**
>> * Converts the double argument to a <code>long</code> using the
>> * <code>doubleToLongBits</code> method in class
>> <code>Double</code>, * and then writes that <code>long</code>
>> value to the underlying * output stream as an 8-byte quantity,
>> high byte first. If no * exception is thrown, the counter
>> <code>written</code> is * incremented by <code>8</code>.
>> *
>> * @param value a <code>double</code> value to be written.
>> * @exception IOException if an I/O error occurs.
>> * @see java.lang.Double#doubleToLongBits(double)
>> */
>> public void writeDouble(double value) throws IOException {
>> writeLong(Double.doubleToLongBits(value));
>> }
>>
>> /**
>> * Writes a <code>char</code> to the underlying output stream as a
>> * 2-byte value, high byte first. If no exception is thrown, the
>> * counter <code>written</code> is incremented by <code>2</code>.
>> *
>> * @param value a <code>char</code> value to be written.
>> * @exception IOException if an I/O error occurs.
>> * @see java.io.FilterOutputStream#out
>> */
>> public void writeChar(int value) throws IOException {
>> super.write(getCharBytes(value));
>> }
>>
>>
>>
>>
>> /**
>> * Writes a <code>boolean</code> to the underlying output stream
>> as * a 1-byte value. The value <code>true</code> is written out
>> as the * value <code>(byte)1</code>; the value <code>false</code>
>> is * written out as the value <code>(byte)0</code>. If no
>> exception is * thrown, the counter <code>written</code> is
>> incremented by * <code>1</code>.
>> *
>> * @param value a <code>boolean</code> value to be written.
>> * @exception IOException if an I/O error occurs.
>> */
>> public void writeBoolean(boolean value) throws IOException {
>> writeByte(value ? 1: 0);
>> }
>>
>> /**
>> * Writes out a <code>byte</code> to the underlying output stream
>> as * a 1-byte value. If no exception is thrown, the counter
>> * <code>written</code> is incremented by <code>1</code>.
>> *
>> * @param value a <code>byte</code> value to be written.
>> * @exception IOException if an I/O error occurs.
>> */
>> public void writeByte(int value) throws IOException {
>> super.write(value);
>> }
>>
>> /**
>> * Writes out the string to the underlying output stream as a
>> * sequence of bytes. Each character in the string is written out, in
>> * sequence, by discarding its high eight bits. If no exception is
>> * thrown, the counter <code>written</code> is incremented by the
>> * length of <code>s</code>.
>> *
>> * @param s a string of bytes to be written.
>> * @exception IOException if an I/O error occurs.
>> * @exception NullPointerException if the <code>s</code> is
>> <code>null</code>.
>> */
>> public void writeBytes(String s) throws IOException {
>> int len = s.length();
>> for (int i = 0; i < len; i++) {
>> super.write((byte) s.charAt(i));
>> }
>> }
>>
>>
>> /**
>> * Writes a string to the underlying output stream as a sequence
>> of * characters. Each character is written to the data output
>> stream as * if by the <code>writeChar</code> method. If no
>> exception is * thrown, the counter <code>written</code> is
>> incremented by twice * the length of <code>s</code>.
>> *
>> * @param s a <code>String</code> value to be written.
>> * @exception IOException if an I/O error occurs.
>> * @see #writeChar(int)
>> * @see #writeBytes(String)
>> */
>> public void writeChars(String s) throws IOException {
>> int len = s.length();
>> for (int i = 0; i < len; i++) {
>> writeChar(s.charAt(i));
>> }
>> }
>>
>> /**
>> * Writes a <code>short</code> to the underlying output stream as two
>> * bytes, high byte first. If no exception is thrown, the counter
>> * <code>written</code> is incremented by <code>2</code>.
>> *
>> * @param value a <code>short</code> to be written.
>> * @exception IOException if an I/O error occurs.
>> */
>> public void writeShort(int value) throws IOException {
>> writeChar(value);
>> }
>>
>> /**
>> * Writes a string to the underlying output stream using
>> * <a
>> href="http://java.sun.com/j2se/1.5.0/docs/api/java/io/DataInput.html#modified-utf-8">modified
>> UTF-8</a>
>> * encoding in a machine-independent manner. * <p>
>> * First, two bytes are written to the output stream as if by the
>> * <code>writeShort</code> method giving the number of bytes to
>> * follow. This value is the number of bytes actually written out,
>> * not the length of the string. Following the length, each
>> character * of the string is output, in sequence, using the
>> modified UTF-8 encoding
>> * for the character. If no exception is thrown, the counter
>> * <code>written</code> is incremented by the total number of *
>> bytes written to the output stream. This will be at least two *
>> plus the length of <code>str</code>, and at most two plus *
>> thrice the length of <code>str</code>.
>> *
>> * @param s a string to be written.
>> * @exception IOException if an I/O error occurs.
>> */
>> public void writeUTF(String s) throws IOException {
>> int len = s.length();
>> int utflen = getUTFLen(s, len);
>>
>> // The byte will store the encoded string lenght (as short)
>> followed
>> // by the stream of encoded characters
>> byte[] b = new byte[utflen + 2];
>>
>> // b[0] and b[1] host the encoded string lenght
>> System.arraycopy(getCharBytes(utflen), 0, b, 0, 2);
>> int count = 2;
>>
>> char c;
>> for (int i = 0; i < len; i++) {
>> c = s.charAt(i);
>>
>> if ((c >= 0x0001) && (c <= 0x007f)) {
>> // one byte
>> b[count++] = (byte) c;
>> } else if ((c == 0x0000) || ((c >= 0x0080) && (c <=
>> 0x07ff))) {
>> // two bytes
>> b[count++] = (byte)(0xc0 | (0x1f & (c >> 6)));
>> b[count++] = (byte)(0x80 | (0x3f & c));
>> } else {
>> // three bytes
>> b[count++] = (byte)(0xe0 | (0x0f & (c >> 12)));
>> b[count++] = (byte)(0x80 | (0x3f & (c >> 6)));
>> b[count++] = (byte)(0x80 | (0x3f & c));
>> }
>> }
>>
>> super.write(b);
>> }
>>
>>
>>
>> /**
>> * Computes the encoded <code>string</code> lengh.
>> * * @param s a <code>string</code> to convert.
>> * @param len The <code>string</code> lenght.
>> * @return The encoded <code>string</code> lenght.
>> * @exception UTFDataFormatException if the encoded
>> <code>string</code>
>> * is longer than <code>65535</code> bytes. */
>> private static int getUTFLen(String s, int len) throws
>> UTFDataFormatException {
>> int utflen = 0;
>> char c;
>>
>> for (int i = 0; i < len; i++) {
>> c = s.charAt(i);
>> if ((c >= 0x0001) && (c <= 0x007f)) {
>> // the character uses one byte only
>> utflen++;
>> } else if ((c == 0x0000) || ((c >= 0x0080) && (c <=
>> 0x07ff))) {
>> // the character uses two bytes
>> utflen += 2;
>> } else {
>> // the character uses three bytes
>> utflen += 3;
>> }
>> }
>>
>> if (utflen > 65535) {
>> throw new UTFDataFormatException(
>> "Encoded string too long: " + utflen + " bytes");
>> }
>>
>> return utflen;
>> }
>>
>>
>>
>>
>> //=======================================================================//
>>
>> // Static methods
>>
>> //=======================================================================//
>>
>>
>> /**
>> * Writes an <code>int</code> to the underlying output stream as four
>> * bytes, high byte first. If no exception is thrown, the counter
>> * <code>written</code> is incremented by <code>4</code>.
>> *
>> * @param value an <code>int</code> to be written.
>> * @exception IOException if an I/O error occurs.
>> * @see java.io.FilterOutputStream#out
>> */
>> public static byte[] getIntBytes(int value) {
>> byte[] b = new byte[4];
>> b[0] = (byte)((value >>> 24) & 0xFF);
>> b[1] = (byte)((value >>> 16) & 0xFF);
>> b[2] = (byte)((value >>> 8) & 0xFF);
>> b[3] = (byte)((value >>> 0) & 0xFF);
>> return b;
>> }
>>
>> /**
>> * Writes a <code>long</code> to the underlying output stream as
>> eight
>> * bytes, high byte first. In no exception is thrown, the counter
>> * <code>written</code> is incremented by <code>8</code>.
>> *
>> * @param value a <code>long</code> to be written.
>> * @exception IOException if an I/O error occurs.
>> * @see java.io.FilterOutputStream#out
>> */
>> public static byte[] getLongBytes(long value) {
>> byte[] b = new byte[8];
>> b[0] = (byte)((int)(value >>> 56) & 0xFF);
>> b[1] = (byte)((int)(value >>> 48) & 0xFF);
>> b[2] = (byte)((int)(value >>> 40) & 0xFF);
>> b[3] = (byte)((int)(value >>> 32) & 0xFF);
>> b[4] = (byte)((int)(value >>> 24) & 0xFF);
>> b[5] = (byte)((int)(value >>> 16) & 0xFF);
>> b[6] = (byte)((int)(value >>> 8) & 0xFF);
>> b[7] = (byte)((int)(value >>> 0) & 0xFF);
>> return b;
>> }
>>
>>
>>
>> /**
>> * Writes a <code>char</code> to the underlying output stream as a
>> * 2-byte value, high byte first. If no exception is thrown, the
>> * counter <code>written</code> is incremented by <code>2</code>.
>> *
>> * @param value a <code>char</code> value to be written.
>> * @exception IOException if an I/O error occurs.
>> * @see java.io.FilterOutputStream#out
>> */
>> public static byte[] getCharBytes(int value) {
>> byte[] b = new byte[2];
>> b[0] = (byte)((value >>> 8) & 0xFF);
>> b[1] = (byte)((value >>> 0) & 0xFF);
>> return b;
>> }
>>
>> /**
>> * Converts the double argument to a <code>long</code> using the
>> * <code>doubleToLongBits</code> method in class
>> <code>Double</code>, * and then writes that <code>long</code>
>> value to the underlying * output stream as an 8-byte quantity,
>> high byte first. If no * exception is thrown, the counter
>> <code>written</code> is * incremented by <code>8</code>.
>> *
>> * @param value a <code>double</code> value to be written.
>> * @exception IOException if an I/O error occurs.
>> * @see java.io.FilterOutputStream#out
>> * @see java.lang.Double#doubleToLongBits(double)
>> */
>> public static byte[] getDoubleBytes(double value) {
>> byte[] b = getLongBytes(Double.doubleToLongBits(value));
>> return b;
>> }
>>
>> } // END DeferredPeriodicOutputStream
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [io] new ThresholdingOutputStream implementation
Posted by Stephen Colebourne <sc...@btopenworld.com>.
Sounds interesting. Can you raise a JIRA call for this (enhancement) and
attach the changes to it? That way we won't lose it. Thanks
Stephen
Michele Mazzucco wrote:
> Hi all,
>
> I've extended the ThresholdingOutputStream class with a new class which
> behaves different from DeferredFileOutputStream:
> - when the stream is closed, the content stored in memory is *always*
> flushed to disk (in DeferredFileOutputStream, instead, if the treshold
> is not reached data is lost)
> - DeferredFileOutputStream maintains data in memory only until the
> treshold value has been reached, then it immediately writes every byte
> to disk. Mine implementation, instead, caches treshold bytes in memory,
> and every time that value is reached (that is, treshold, 2 * threshold,
> etc), it flushes data to disk. In other words it acts as a cache.
>
> Please find attached the class together with the unit test.
>
>
>
> Best regards,
> Michele
>
>
> ------------------------------------------------------------------------
>
> /*
> * Licensed to the Apache Software Foundation (ASF) under one or more
> * contributor license agreements. See the NOTICE file distributed with
> * this work for additional information regarding copyright ownership.
> * The ASF licenses this file to You under the Apache License, Version 2.0
> * (the "License"); you may not use this file except in compliance with
> * the License. You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
>
> package org.apache.commons.io.output;
>
> import java.io.DataInputStream;
> import java.io.File;
> import java.io.FileInputStream;
> import java.io.FileNotFoundException;
> import java.io.IOException;
> import java.util.Arrays;
>
> import junit.framework.TestCase;
>
> /**
> * JUnit test case for {@link DeferredPeriodicOutputStream}.
> *
> * @author <a href="mailto:Michele.Mazzucco@ncl.ac.uk">Michele Mazzucco</a>
> *
> */
> public class DeferredPeriodicOutputStreamTest extends TestCase {
>
>
> /**
> * The test data as a string (which is the simplest form).
> */
> private String testString = "0123456789";
>
> /**
> * The test data as a byte array, derived from the string.
> */
> private byte[] testBytes = testString.getBytes();
>
>
> /**
> * Standard JUnit test case constructor.
> *
> * @param name
> * The name of the test case.
> */
> public DeferredPeriodicOutputStreamTest(String name) {
> super(name);
> }
>
>
> /**
> * Tests the case where the amount of data falls below the threshold, and is
> * therefore confined to memory.
> */
> public void testBelowThreshold() {
> File testFile = getTestFile("testBelowTreshold.dat");
>
> DeferredPeriodicOutputStream dpos =
> new DeferredPeriodicOutputStream(testBytes.length + 42, testFile);
> try {
> dpos.write(testBytes, 0, testBytes.length);
> dpos.close();
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> byte[] resultBytes = dpos.getData();
> assertEquals(0, resultBytes.length);
>
> verifyResultFile(testFile);
> }
>
>
> /**
> * Tests the case where the amount of data is exactly the same as the
> * threshold. The behavior should be the same as that for the amount of
> * data being below (i.e. not exceeding) the threshold.
> */
> public void testAtThreshold() {
> File testFile = getTestFile("testAtThreshold.dat");
>
> DeferredPeriodicOutputStream dpos =
> new DeferredPeriodicOutputStream(testBytes.length, testFile);
> try
> {
> dpos.write(testBytes, 0, testBytes.length);
> dpos.close();
> }
> catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> byte[] resultBytes = dpos.getData();
> assertEquals(0, resultBytes.length);
> verifyResultFile(testFile);
> }
>
> /**
> * Tests the case where the amount of data exceeds the threshold, and is
> * therefore written to disk. The actual data written to disk is verified,
> * as is the file itself.
> */
> public void testAboveThreshold() {
> File testFile = getTestFile("testAboveThreshold.dat");
>
> DeferredFileOutputStream dfos =
> new DeferredFileOutputStream(testBytes.length - 5, testFile);
> try
> {
> dfos.write(testBytes, 0, testBytes.length);
> dfos.close();
> }
> catch (IOException e) {
> fail("Unexpected IOException");
> }
> assertFalse(dfos.isInMemory());
> assertNull(dfos.getData());
>
> verifyResultFile(testFile);
> }
>
> /**
> * Verifies that the specified file contains the same data as the original
> * test data.
> *
> * @param testFile The file containing the test output.
> */
> private void verifyResultFile(File testFile) {
> try
> {
> FileInputStream fis = new FileInputStream(testFile);
> assertTrue(fis.available() == testBytes.length);
>
> byte[] resultBytes = new byte[testBytes.length];
> assertTrue(fis.read(resultBytes) == testBytes.length);
>
> assertTrue(Arrays.equals(resultBytes, testBytes));
> assertTrue(fis.read(resultBytes) == -1);
>
> try
> {
> fis.close();
> }
> catch (IOException e) {
> // Ignore an exception on close
> }
> }
> catch (FileNotFoundException e) {
> fail("Unexpected FileNotFoundException");
> }
> catch (IOException e) {
> fail("Unexpected IOException");
> }
> }
>
>
> public void testCheckThreshold() {
> File testFile = getTestFile("testCheckThreshold.dat");
>
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(8, testFile);
>
> try {
> dpos.write(testBytes);
> dpos.close();
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
> assertTrue(dpos.isThresholdExceeded());
> }
>
> private static File getTestFile(String path) {
> File testFile = new File(path);
> testFile.deleteOnExit();
> testFile.delete();
>
> return testFile;
> }
>
> public void testGetStream() {
> File testFile = getTestFile("testGetStream.dat");
>
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(12, testFile);
> try {
> dpos.write(testBytes);
> byte[] memory = dpos.getData();
> assertEquals(memory.length, 10);
> assertTrue(Arrays.equals(testBytes, memory));
>
> dpos.close();
> memory = dpos.getData();
> assertEquals(memory.length, 0);
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
> }
>
> /**
> * Tests the write of a int value.
> */
> public void testWriteInt() {
> File testFile = getTestFile("testWriteInt.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(5, testFile);
>
> try {
> dpos.writeInt(5);
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> int read = getInt(dpos.getData());
> assertTrue(read == 5);
> }
>
> /**
> * Tests the write of a float value.
> */
> public void testWriteFloat() {
> File testFile = getTestFile("testWriteFloat.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(9, testFile);
>
> try {
> dpos.writeFloat(15.65f);
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> float read = getFloat(dpos.getData());
> assertTrue(read == 15.65f);
> }
>
> /**
> * Tests the write of a long value.
> */
> public void testWriteLong() {
> File testFile = getTestFile("testWriteLong.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(9, testFile);
>
> long value = System.currentTimeMillis();
> try {
> dpos.writeLong(value);
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> float read = getLong(dpos.getData());
> assertTrue(read == value);
> }
>
> /**
> * Tests the write of a double value.
> */
> public void testWriteDouble() {
> File testFile = getTestFile("testWriteDouble.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(9, testFile);
>
> try {
> dpos.writeDouble(Math.PI);
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> double read = getDouble(dpos.getData());
> assertTrue(read == Math.PI);
> }
>
>
> /**
> * Tests the write of a char value.
> */
> public void testWriteChar() {
> File testFile = getTestFile("testWriteChar.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(3, testFile);
>
> try {
> dpos.writeChar('b');
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> char c = getChar(dpos.getData());
> assertTrue(c == 'b');
> }
>
> /**
> * Tests the write of a boolean value.
> */
> public void testWriteBoolean() {
> File testFile = getTestFile("testWriteBoolean.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(2, testFile);
>
> try {
> dpos.writeBoolean(true);
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> boolean read = (dpos.getData()[0] != 0);
> assertTrue(read == true);
> }
>
>
> /**
> * Tests the write of a byte value.
> */
> public void testWriteByte() {
> File testFile = getTestFile("testWriteByte.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(2, testFile);
>
> try {
> dpos.writeByte(3);
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> byte read = (byte) (dpos.getData()[0]);
> assertTrue(read == 3);
> }
>
> public void testWriteBytes() {
> File testFile = getTestFile("testWriteBytes.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(2, testFile);
>
> try {
> dpos.writeBytes(testString);
> dpos.close();
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> verifyResultFile(testFile);
> }
>
> public void testWriteChars() {
> File testFile = getTestFile("testWriteChars.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(21, testFile);
>
> try {
> dpos.writeChars(testString);
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> byte[] read = dpos.getData();
> char[] c = new char[10];
> byte[] b = new byte[2];
> int j = 0;
> for (int i = 0; i < read.length; i = (i + 2)) {
> System.arraycopy(read, i, b, 0, 2);
> c[j] = getChar(b);
> j++;
> }
>
> assertEquals(testString, new String(c));
> }
>
> /**
> * Tests the write of a short value.
> */
> public void testWriteShort() {
> File testFile = getTestFile("testWriteShort.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(3, testFile);
>
> try {
> dpos.writeShort(3);
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> short read = (short) getChar(dpos.getData());
> assertTrue(read == 3);
> }
>
> /**
> * Tests the write of an UTF string.
> */
> public void testWriteUTF() {
> File testFile = getTestFile("testWriteUTF.dat");
> DeferredPeriodicOutputStream dpos = new DeferredPeriodicOutputStream(3, testFile);
>
> try {
> dpos.writeUTF(testString);
> dpos.close();
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
>
> try {
> DataInputStream dis = new DataInputStream(new FileInputStream(testFile));
> String s = dis.readUTF();
>
> assertEquals(testString, s);
>
> try {
> dis.close();
> } catch (IOException e) {
> // Ignore this
> }
> } catch (FileNotFoundException e) {
> fail("Unexpected FileNotFoundException");
> } catch (IOException e) {
> fail("Unexpected IOException");
> }
> }
>
>
>
> //===================================================================//
> // Static methods used to convert array of bytes to primitive values
> //===================================================================//
>
> /**
> * Converts the specified array of <code>bytes</code> to a <code>int</code>.
> *
> * @param b An array of <code>bytes</code>.
> * @return The equivalent <code>int</code> value.
> */
> public static int getInt(byte[] b) {
> assert b.length == 4: "Invalid number of bytes for integer conversion";
> return ((b[0] << 24) & 0xFF000000) + ((b[1] << 16) & 0x00FF0000) +
> ((b[2] << 8) & 0x0000FF00) + ((b[3] << 0) & 0x000000FF);
> }
>
> /**
> * Converts the specified array of <code>bytes</code> to a <code>float</code>.
> *
> * @param b An array of <code>bytes</code>.
> * @return The equivalent <code>float</code> value.
> */
> public static float getFloat(byte[] b) {
> assert b.length == 4: "Invalid number of bytes for float conversion";
> return Float.intBitsToFloat(getInt(b));
> }
>
> /**
> * Converts the specified array of <code>bytes</code> to a <code>long</code>.
> *
> * @param b An array of <code>bytes</code>.
> * @return The equivalent <code>long</code> value.
> */
> public static long getLong(byte[] b) {
> assert b.length == 8: "Invalid number of bytes for long conversion";
> int high = getInt(new byte[] { b[0], b[1], b[2], b[3] });
> int low = getInt(new byte[] { b[4], b[5], b[6], b[7] });
> long value = ((long)(high) << 32) + (low & 0xFFFFFFFFL);
> return value;
> }
>
> /**
> * Converts the specified array of <code>bytes</code> to a <code>double</code>.
> *
> * @param b An array of <code>bytes</code>.
> * @return The equivalent <code>double</code> value.
> */
> public static double getDouble(byte[] b) {
> assert b.length == 8: "Invalid number of bytes for double conversion";
> return Double.longBitsToDouble(getLong(b));
> }
>
> /**
> * Converts the specified array of <code>bytes</code> to a <code>char</code>.
> *
> * @param b An array of <code>bytes</code>.
> * @return The equivalent <code>char</code> value.
> */
> public static char getChar(byte[] b) {
> assert b.length == 2: "Invalid number of bytes for char conversion";
> return (char)(((b[0] << 8) & 0x0000FF00) + ((b[1] << 0) & 0x000000FF));
> }
> }
>
>
> ------------------------------------------------------------------------
>
> /*
> * Licensed to the Apache Software Foundation (ASF) under one or more
> * contributor license agreements. See the NOTICE file distributed with
> * this work for additional information regarding copyright ownership.
> * The ASF licenses this file to You under the Apache License, Version 2.0
> * (the "License"); you may not use this file except in compliance with
> * the License. You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
>
> package org.apache.commons.io.output;
>
> import java.io.DataOutput;
> import java.io.File;
> import java.io.FileOutputStream;
> import java.io.IOException;
> import java.io.OutputStream;
> import java.io.UTFDataFormatException;
>
>
> /**
> * The <code>DeferredPeriodicOutputStream</code> is an output stream which
> * retains data in memory until a specified treshold is reached, commit it
> * to disk and reset the threshold value. The result is that data are written
> * to disk on a periodic basis, that is, every <i>treshold</i> bytes. In other
> * words, this output stream acts as a cache.
> * <p>
> * Opposite to {@link DeferredFileOutputStream}, if the stream is
> * closed before the threshold is reached, the data are written to
> * disk.
> *
> * @author <a href="mailto:Michele.Mazzucco@ncl.ac.uk">Michele Mazzucco</a>
> * @version $Version$
> * @see org.apache.commons.io.output.DeferredFileOutputStream
> */
> public class DeferredPeriodicOutputStream extends ThresholdingOutputStream
> implements DataOutput {
>
>
> //=======================================================================//
> // Instance fields
> //=======================================================================//
>
>
> /**
> * The output stream to which data will be written before every theshold
> * is reached.
> */
> private ByteArrayOutputStream memoryOutputStream;
>
> /**
> * The output stream to which data will be written after every theshold
> * is reached.
> */
> private FileOutputStream diskOutputStream;
>
> /**
> * The file to which output will be directed every time the threshold is
> * exceeded.
> */
> private File outputFile;
>
>
>
> //=======================================================================//
> // Constructor
> //=======================================================================//
>
>
> /**
> * Creates a new <tt>DeferredPeriodicOutputStream</tt> object.
> *
> * @param threshold The number of bytes to keep in memory.
> * @param file The file to use every time the threshold is reached.
> */
> public DeferredPeriodicOutputStream(int threshold, File file) {
> super(threshold);
> this.outputFile = file;
> this.memoryOutputStream = new ByteArrayOutputStream(threshold);
> }
>
> /**
> * Creates a new <tt>DeferredPeriodicOutputStream</tt> object.
> *
> * @param threshold The number of bytes to keep in memory.
> * @param file The path to the file to use every time the threshold
> * is reached.
> */
> public DeferredPeriodicOutputStream(int treshold, String path) {
> this(treshold, new File(path));
> }
>
>
> //=======================================================================//
> // Methods from ThresholdingOutputStream
> //=======================================================================//
>
>
> /**
> * Gets the memory output stream (<i>always</i>).
> *
> * @return The memory ouput stream.
> */
> @Override
> protected OutputStream getStream() {
> return memoryOutputStream;
> }
>
> @Override
> protected synchronized void thresholdReached() throws IOException {
> if (diskOutputStream == null) {
> diskOutputStream = new FileOutputStream(outputFile);
> }
> memoryOutputStream.writeTo(diskOutputStream);
> diskOutputStream.flush();
> memoryOutputStream.reset();
> }
>
>
> @Override
> protected synchronized void checkThreshold(int count) throws IOException {
> if ((memoryOutputStream.size() + count) > super.getThreshold()) {
> this.thresholdReached();
> }
> }
>
> /**
> * Closes this output stream flushing all cached data to disk.
> */
> @Override
> public void close() throws IOException {
> super.close();
> if (diskOutputStream == null) {
> if (memoryOutputStream.size() > 0) {
> diskOutputStream = new FileOutputStream(outputFile);
> } else {
> return;
> }
> }
>
> memoryOutputStream.writeTo(diskOutputStream);
> flush();
> memoryOutputStream.reset();
> memoryOutputStream.close();
>
> if (diskOutputStream != null) {
> diskOutputStream.close();
> }
> }
>
>
> /**
> * Flushes this output stream and forces any buffered output bytes to be
> * written out.
> *
> * @exception IOException if an error occurs.
> */
> @Override
> public void flush() throws IOException {
> memoryOutputStream.flush();
>
> if (diskOutputStream != null)
> diskOutputStream.flush();
> }
>
> // ======================================================================//
> // Public methods
> //=======================================================================//
>
>
>
> /**
> * Returns an array of bytes containing between <code>0</code> and
> * {@link ThresholdingOutputStream#threshold} bytes. The array corresponds
> * to the data cached in memory.
> *
> * @return The data for this output stream which are maintained in memory.
> */
> public byte[] getData() {
> return memoryOutputStream.toByteArray();
> }
>
>
> //=======================================================================//
> // Methods from java.io.DataOuput
> //=======================================================================//
>
>
> /**
> * Writes an <code>int</code> to the underlying output stream as four
> * bytes, high byte first. If no exception is thrown, the counter
> * <code>written</code> is incremented by <code>4</code>.
> *
> * @param value an <code>int</code> to be written.
> * @exception IOException if an I/O error occurs.
> */
> public void writeInt(int value) throws IOException {
> super.write(getIntBytes(value));
> }
>
> /**
> * Converts the float argument to an <code>int</code> using the
> * <code>floatToIntBits</code> method in class <code>Float</code>,
> * and then writes that <code>int</code> value to the underlying
> * output stream as a 4-byte quantity, high byte first. If no
> * exception is thrown, the counter <code>written</code> is
> * incremented by <code>4</code>.
> *
> * @param value a <code>float</code> value to be written.
> * @exception IOException if an I/O error occurs.
> * @see java.lang.Float#floatToIntBits(float)
> */
> public void writeFloat(float value) throws IOException {
> writeInt(Float.floatToIntBits(value));
> }
>
> /**
> * Writes a <code>long</code> to the underlying output stream as eight
> * bytes, high byte first. In no exception is thrown, the counter
> * <code>written</code> is incremented by <code>8</code>.
> *
> * @param value a <code>long</code> to be written.
> * @exception IOException if an I/O error occurs.
> * @see java.io.FilterOutputStream#out
> */
> public void writeLong(long value) throws IOException {
> super.write(getLongBytes(value));
> }
>
> /**
> * Converts the double argument to a <code>long</code> using the
> * <code>doubleToLongBits</code> method in class <code>Double</code>,
> * and then writes that <code>long</code> value to the underlying
> * output stream as an 8-byte quantity, high byte first. If no
> * exception is thrown, the counter <code>written</code> is
> * incremented by <code>8</code>.
> *
> * @param value a <code>double</code> value to be written.
> * @exception IOException if an I/O error occurs.
> * @see java.lang.Double#doubleToLongBits(double)
> */
> public void writeDouble(double value) throws IOException {
> writeLong(Double.doubleToLongBits(value));
> }
>
> /**
> * Writes a <code>char</code> to the underlying output stream as a
> * 2-byte value, high byte first. If no exception is thrown, the
> * counter <code>written</code> is incremented by <code>2</code>.
> *
> * @param value a <code>char</code> value to be written.
> * @exception IOException if an I/O error occurs.
> * @see java.io.FilterOutputStream#out
> */
> public void writeChar(int value) throws IOException {
> super.write(getCharBytes(value));
> }
>
>
>
>
>
> /**
> * Writes a <code>boolean</code> to the underlying output stream as
> * a 1-byte value. The value <code>true</code> is written out as the
> * value <code>(byte)1</code>; the value <code>false</code> is
> * written out as the value <code>(byte)0</code>. If no exception is
> * thrown, the counter <code>written</code> is incremented by
> * <code>1</code>.
> *
> * @param value a <code>boolean</code> value to be written.
> * @exception IOException if an I/O error occurs.
> */
> public void writeBoolean(boolean value) throws IOException {
> writeByte(value ? 1: 0);
> }
>
> /**
> * Writes out a <code>byte</code> to the underlying output stream as
> * a 1-byte value. If no exception is thrown, the counter
> * <code>written</code> is incremented by <code>1</code>.
> *
> * @param value a <code>byte</code> value to be written.
> * @exception IOException if an I/O error occurs.
> */
> public void writeByte(int value) throws IOException {
> super.write(value);
> }
>
> /**
> * Writes out the string to the underlying output stream as a
> * sequence of bytes. Each character in the string is written out, in
> * sequence, by discarding its high eight bits. If no exception is
> * thrown, the counter <code>written</code> is incremented by the
> * length of <code>s</code>.
> *
> * @param s a string of bytes to be written.
> * @exception IOException if an I/O error occurs.
> * @exception NullPointerException if the <code>s</code> is <code>null</code>.
> */
> public void writeBytes(String s) throws IOException {
> int len = s.length();
> for (int i = 0; i < len; i++) {
> super.write((byte) s.charAt(i));
> }
> }
>
>
> /**
> * Writes a string to the underlying output stream as a sequence of
> * characters. Each character is written to the data output stream as
> * if by the <code>writeChar</code> method. If no exception is
> * thrown, the counter <code>written</code> is incremented by twice
> * the length of <code>s</code>.
> *
> * @param s a <code>String</code> value to be written.
> * @exception IOException if an I/O error occurs.
> * @see #writeChar(int)
> * @see #writeBytes(String)
> */
> public void writeChars(String s) throws IOException {
> int len = s.length();
> for (int i = 0; i < len; i++) {
> writeChar(s.charAt(i));
> }
> }
>
> /**
> * Writes a <code>short</code> to the underlying output stream as two
> * bytes, high byte first. If no exception is thrown, the counter
> * <code>written</code> is incremented by <code>2</code>.
> *
> * @param value a <code>short</code> to be written.
> * @exception IOException if an I/O error occurs.
> */
> public void writeShort(int value) throws IOException {
> writeChar(value);
> }
>
> /**
> * Writes a string to the underlying output stream using
> * <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/io/DataInput.html#modified-utf-8">modified UTF-8</a>
> * encoding in a machine-independent manner.
> * <p>
> * First, two bytes are written to the output stream as if by the
> * <code>writeShort</code> method giving the number of bytes to
> * follow. This value is the number of bytes actually written out,
> * not the length of the string. Following the length, each character
> * of the string is output, in sequence, using the modified UTF-8 encoding
> * for the character. If no exception is thrown, the counter
> * <code>written</code> is incremented by the total number of
> * bytes written to the output stream. This will be at least two
> * plus the length of <code>str</code>, and at most two plus
> * thrice the length of <code>str</code>.
> *
> * @param s a string to be written.
> * @exception IOException if an I/O error occurs.
> */
> public void writeUTF(String s) throws IOException {
> int len = s.length();
> int utflen = getUTFLen(s, len);
>
> // The byte will store the encoded string lenght (as short) followed
> // by the stream of encoded characters
> byte[] b = new byte[utflen + 2];
>
> // b[0] and b[1] host the encoded string lenght
> System.arraycopy(getCharBytes(utflen), 0, b, 0, 2);
> int count = 2;
>
> char c;
> for (int i = 0; i < len; i++) {
> c = s.charAt(i);
>
> if ((c >= 0x0001) && (c <= 0x007f)) {
> // one byte
> b[count++] = (byte) c;
> } else if ((c == 0x0000) || ((c >= 0x0080) && (c <= 0x07ff))) {
> // two bytes
> b[count++] = (byte)(0xc0 | (0x1f & (c >> 6)));
> b[count++] = (byte)(0x80 | (0x3f & c));
> } else {
> // three bytes
> b[count++] = (byte)(0xe0 | (0x0f & (c >> 12)));
> b[count++] = (byte)(0x80 | (0x3f & (c >> 6)));
> b[count++] = (byte)(0x80 | (0x3f & c));
> }
> }
>
> super.write(b);
> }
>
>
>
> /**
> * Computes the encoded <code>string</code> lengh.
> *
> * @param s a <code>string</code> to convert.
> * @param len The <code>string</code> lenght.
> * @return The encoded <code>string</code> lenght.
> * @exception UTFDataFormatException if the encoded <code>string</code>
> * is longer than <code>65535</code> bytes.
> */
> private static int getUTFLen(String s, int len) throws UTFDataFormatException {
> int utflen = 0;
> char c;
>
> for (int i = 0; i < len; i++) {
> c = s.charAt(i);
> if ((c >= 0x0001) && (c <= 0x007f)) {
> // the character uses one byte only
> utflen++;
> } else if ((c == 0x0000) || ((c >= 0x0080) && (c <= 0x07ff))) {
> // the character uses two bytes
> utflen += 2;
> } else {
> // the character uses three bytes
> utflen += 3;
> }
> }
>
> if (utflen > 65535) {
> throw new UTFDataFormatException(
> "Encoded string too long: " + utflen + " bytes");
> }
>
> return utflen;
> }
>
>
>
> //=======================================================================//
> // Static methods
> //=======================================================================//
>
> /**
> * Writes an <code>int</code> to the underlying output stream as four
> * bytes, high byte first. If no exception is thrown, the counter
> * <code>written</code> is incremented by <code>4</code>.
> *
> * @param value an <code>int</code> to be written.
> * @exception IOException if an I/O error occurs.
> * @see java.io.FilterOutputStream#out
> */
> public static byte[] getIntBytes(int value) {
> byte[] b = new byte[4];
> b[0] = (byte)((value >>> 24) & 0xFF);
> b[1] = (byte)((value >>> 16) & 0xFF);
> b[2] = (byte)((value >>> 8) & 0xFF);
> b[3] = (byte)((value >>> 0) & 0xFF);
> return b;
> }
>
> /**
> * Writes a <code>long</code> to the underlying output stream as eight
> * bytes, high byte first. In no exception is thrown, the counter
> * <code>written</code> is incremented by <code>8</code>.
> *
> * @param value a <code>long</code> to be written.
> * @exception IOException if an I/O error occurs.
> * @see java.io.FilterOutputStream#out
> */
> public static byte[] getLongBytes(long value) {
> byte[] b = new byte[8];
> b[0] = (byte)((int)(value >>> 56) & 0xFF);
> b[1] = (byte)((int)(value >>> 48) & 0xFF);
> b[2] = (byte)((int)(value >>> 40) & 0xFF);
> b[3] = (byte)((int)(value >>> 32) & 0xFF);
> b[4] = (byte)((int)(value >>> 24) & 0xFF);
> b[5] = (byte)((int)(value >>> 16) & 0xFF);
> b[6] = (byte)((int)(value >>> 8) & 0xFF);
> b[7] = (byte)((int)(value >>> 0) & 0xFF);
> return b;
> }
>
>
>
> /**
> * Writes a <code>char</code> to the underlying output stream as a
> * 2-byte value, high byte first. If no exception is thrown, the
> * counter <code>written</code> is incremented by <code>2</code>.
> *
> * @param value a <code>char</code> value to be written.
> * @exception IOException if an I/O error occurs.
> * @see java.io.FilterOutputStream#out
> */
> public static byte[] getCharBytes(int value) {
> byte[] b = new byte[2];
> b[0] = (byte)((value >>> 8) & 0xFF);
> b[1] = (byte)((value >>> 0) & 0xFF);
> return b;
> }
>
> /**
> * Converts the double argument to a <code>long</code> using the
> * <code>doubleToLongBits</code> method in class <code>Double</code>,
> * and then writes that <code>long</code> value to the underlying
> * output stream as an 8-byte quantity, high byte first. If no
> * exception is thrown, the counter <code>written</code> is
> * incremented by <code>8</code>.
> *
> * @param value a <code>double</code> value to be written.
> * @exception IOException if an I/O error occurs.
> * @see java.io.FilterOutputStream#out
> * @see java.lang.Double#doubleToLongBits(double)
> */
> public static byte[] getDoubleBytes(double value) {
> byte[] b = getLongBytes(Double.doubleToLongBits(value));
> return b;
> }
>
> } // END DeferredPeriodicOutputStream
>
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org