You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by "Neiko2002 (via GitHub)" <gi...@apache.org> on 2024/04/17 16:37:56 UTC

[I] [Java] serializeJavaObject(...) is faster than serialize(...) [incubator-fury]

Neiko2002 opened a new issue, #1537:
URL: https://github.com/apache/incubator-fury/issues/1537

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-fury/issues) and found no similar issues.
   
   
   ### Version
   
   Version: 0.5.0-SNAPSHOT
   OS: Windows
   JDK: 22
   
   ### Component(s)
   
   Java
   
   ### Minimal reproduce step
   
   ```
   public static void main(String[] args) throws Exception {
   	final byte[] data = new byte[406991872];
   
   	int runs = 5;
   	Fury fury = Fury.builder().withLanguage(Language.JAVA)
   			.withRefTracking(false)
   			.requireClassRegistration(true)
   			.withNumberCompressed(false)
   			.withStringCompressed(false)
   			.build();
   
   	{
   		try (Output output = new Output(Files.newOutputStream(furyFile), 1024*1024)) {
   			fury.serializeJavaObject(output, data);
   		}			
   		long start = System.currentTimeMillis();
   		for (int i = 0; i < runs; i++) {	
   			try (Output output = new Output(Files.newOutputStream(furyFile), 1024*1024)) {
   				fury.serializeJavaObject(output, data); // warmup
   			}
   		}
   		System.out.println("serializeJavaObject took "+(System.currentTimeMillis()-start)+"ms");
   	}
   
   	{
   		try (Output output = new Output(Files.newOutputStream(furyFile), 1024*1024)) {
   			fury.serialize(output, data); // warmup
   		}			
   		long start = System.currentTimeMillis();
   		for (int i = 0; i < runs; i++) {				
   			try (Output output = new Output(Files.newOutputStream(furyFile), 1024*1024)) {
   			fury.serialize(output, data);
   			}
   		}
   		System.out.println("serialize took "+(System.currentTimeMillis()-start)+"ms");
   	}
   }
   ```
   
   ### What did you expect to see?
   
   I would expect the serializeJavaObject(...) runs a little bit slower because of the additional class information which need to be stored, but the method is actually faster than serialize(...). In this example its only 10% difference. For a more complex project we get 25% faster serialization times for serializeJavaObject(...) than serialize(...)
   
   ### What did you see instead?
   
   ```
   18:26:27.900 [main] INFO org.apache.fury.Fury -- Created new fury org.apache.fury.Fury@52aa2946
   serializeJavaObject took 7352ms
   serialize took 7849ms
   ```
   
   
   ### Anything Else?
   
   This is not a perfect benchmark (no JMH), but no matter how many additional runs we add to the test or if we change the execution order the results are always the same.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org


Re: [I] [Java] serializeJavaObject(...) is faster than serialize(...) [incubator-fury]

Posted by "heliang666s (via GitHub)" <gi...@apache.org>.
heliang666s commented on issue #1537:
URL: https://github.com/apache/incubator-fury/issues/1537#issuecomment-2063645369

   The difference between the two is very small, and it is likely that it is due to error. I copied your code and tested it, and the object was a 2M size image. The number of cycles was increased in turn, and it was found that there was not much difference between the two.When cycles is small,'serializeJavaObject' is faster,but But when the number of cycles is large, the situation is reversed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org


Re: [I] [Java] serializeJavaObject(...) is faster than serialize(...) [fory]

Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1537:
URL: https://github.com/apache/fory/issues/1537#issuecomment-4418440972

   serializeJavaObject has been removed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org


Re: [I] [Java] serializeJavaObject(...) is faster than serialize(...) [incubator-fury]

Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1537:
URL: https://github.com/apache/incubator-fury/issues/1537#issuecomment-2063626722

   `serializeJavaObject` won't store classinfo of outermost object, so it would be faster. But The difference between `serializeJavaObject` and `serialize` should be small, since it only save a classinto write for passed object type


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org


Re: [I] [Java] serializeJavaObject(...) is faster than serialize(...) [fory]

Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang closed issue #1537: [Java] serializeJavaObject(...) is faster than serialize(...)
URL: https://github.com/apache/fory/issues/1537


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fory.apache.org
For additional commands, e-mail: commits-help@fory.apache.org


Re: [I] [Java] serializeJavaObject(...) is faster than serialize(...) [incubator-fury]

Posted by "Neiko2002 (via GitHub)" <gi...@apache.org>.
Neiko2002 commented on issue #1537:
URL: https://github.com/apache/incubator-fury/issues/1537#issuecomment-2074584255

   In my original question, I used a class not included in the Java JDK, making it hard for others to understand the behavior. Unfortunately, the BufferedOutputStream class from the JDK is very slow in this use case, which is why I used the Output class from Kryo. Below, I provide a standalone program without external dependencies, where I've copied the important parts of the Kryo Output class.
   
   ```
   import java.io.IOException;
   import java.io.OutputStream;
   import java.nio.file.Files;
   import java.nio.file.Path;
   import java.nio.file.Paths;
   
   import org.apache.fury.Fury;
   import org.apache.fury.config.Language;
   
   public class FuryRegressionBench {
   
   	public static final Path furyFile = Paths.get("test.fury");
   	
   	public static final int warumup = 3;
   	public static final int runs = 5;
   	public static final byte[] data = new byte[406991872];
   
   	public static void main(String[] args) throws Exception {
   
   		Fury fury = Fury.builder().withLanguage(Language.JAVA)
   				.withRefTracking(false)
   				.requireClassRegistration(true)
   				.withNumberCompressed(false)
   				.withStringCompressed(false)
   				.build();
   		
   		interface SerializeFunc {
   			public void execute(OutputStream output) throws IOException; 
   			
   			public static void run(String funcName, SerializeFunc func)  {
   				try {			
   					for (int i = 0; i < warumup; i++) {				
   						try (OutputStream output = new BufferedOutput(Files.newOutputStream(furyFile), 1024*1024)) {
   							func.execute(output);
   						}
   					}
   					
   					long start = System.currentTimeMillis();
   					for (int i = 0; i < runs; i++) {				
   						try (OutputStream output = new BufferedOutput(Files.newOutputStream(furyFile), 1024*1024)) {
   							func.execute(output);
   						}
   					}
   					System.out.println(funcName+" took "+(System.currentTimeMillis()-start)+"ms");
   				} catch (Exception e) {
   					e.printStackTrace();
   				}
   			}
   		}
   
   		SerializeFunc.run("BufferedOutput", (output) -> output.write(data));
   		SerializeFunc.run("serializeJavaObject", (output) -> fury.serializeJavaObject(output, data));
   		SerializeFunc.run("serialize", (output) -> fury.serialize(output, data));
   	}
   	
   	/**
   	 * Extract of kryo Output.java
   	 */
   	public static class BufferedOutput extends OutputStream implements AutoCloseable {
   
   		// Maximum reasonable array length. See: https://stackoverflow.com/questions/3038392/do-java-arrays-have-a-maximum-size
   		public static final int maxArraySize = Integer.MAX_VALUE - 8;
   
   		private int maxCapacity;
   		private int position;
   		private int capacity;
   		private byte[] buffer;
   		private OutputStream outputStream;
   
   		/** Creates a new Output for writing to a byte[].
   		 * @param bufferSize The initial size of the buffer.
   		 * @param maxBufferSize If {@link #flush()} does not empty the buffer, the buffer is doubled as needed until it exceeds
   		 *           maxBufferSize and an exception is thrown. Can be -1 for no maximum. */
   		public BufferedOutput (int bufferSize, int maxBufferSize) {
   			if (bufferSize > maxBufferSize && maxBufferSize != -1) throw new IllegalArgumentException(
   					"bufferSize: " + bufferSize + " cannot be greater than maxBufferSize: " + maxBufferSize);
   			if (maxBufferSize < -1) throw new IllegalArgumentException("maxBufferSize cannot be < -1: " + maxBufferSize);
   			this.capacity = bufferSize;
   			this.maxCapacity = maxBufferSize == -1 ? maxArraySize : maxBufferSize;
   			buffer = new byte[bufferSize];
   		}
   
   		/** Creates a new Output for writing to an OutputStream with the specified buffer size. */
   		public BufferedOutput (OutputStream outputStream, int bufferSize) {
   			this(bufferSize, bufferSize);
   			if (outputStream == null) throw new IllegalArgumentException("outputStream cannot be null.");
   			this.outputStream = outputStream;
   		}
   
   		/** Flushes the buffered bytes. The default implementation writes the buffered bytes to the {@link #getOutputStream()
   		 * OutputStream}, if any, and sets the position to 0. Can be overridden to flush the bytes somewhere else. */
   		public void flush() throws IOException {
   			if (outputStream == null) return;
   			try {
   				outputStream.write(buffer, 0, position);
   				outputStream.flush();
   			} catch (IOException ex) {
   				throw new IOException(ex);
   			}
   			position = 0;
   		}
   
   		/** Ensures the buffer is large enough to read the specified number of bytes.
   		 * @return true if the buffer has been resized. */
   		protected boolean require(int required) throws IOException {
   			if (capacity - position >= required) return false;
   			flush();
   			if (capacity - position >= required) return true;
   			if (required > maxCapacity - position) {
   				if (required > maxCapacity)
   					throw new IOException("Buffer overflow. Max capacity: " + maxCapacity + ", required: " + required);
   				throw new IOException(
   						"Buffer overflow. Available: " + (maxCapacity - position) + ", required: " + required);
   			}
   			if (capacity == 0) capacity = 16;
   			do {
   				capacity = Math.min(capacity * 2, maxCapacity);
   			} while (capacity - position < required);
   			byte[] newBuffer = new byte[capacity];
   			System.arraycopy(buffer, 0, newBuffer, 0, position);
   			buffer = newBuffer;
   			return true;
   		}
   
   		@Override
   		public void write(int value) throws IOException {
   			if (position == capacity) require(1);
   			buffer[position++] = (byte)value;
   		}		
   
   		/** Writes the bytes. Note the number of bytes is not written. */
   		public void write(byte[] bytes) throws IOException {
   			if (bytes == null) throw new IllegalArgumentException("bytes cannot be null.");
   			writeBytes(bytes, 0, bytes.length);
   		}
   
   		/** Writes the bytes. Note the number of bytes is not written. */
   		public void write(byte[] bytes, int offset, int length) throws IOException {
   			writeBytes(bytes, offset, length);
   		}
   
   		/** Writes the bytes. Note the number of bytes is not written. */
   		public void writeBytes(byte[] bytes, int offset, int count) throws IOException {
   			if (bytes == null) throw new IllegalArgumentException("bytes cannot be null.");
   			int copyCount = Math.min(capacity - position, count);
   			while (true) {
   				System.arraycopy(bytes, offset, buffer, position, copyCount);
   				position += copyCount;
   				count -= copyCount;
   				if (count == 0) return;
   				offset += copyCount;
   				copyCount = Math.min(Math.max(capacity, 1), count);
   				require(copyCount);
   			}
   		}
   
   		/** Flushes any buffered bytes and closes the underlying OutputStream, if any. */
   		@Override
   		public void close() throws IOException {
   			flush();
   			if (outputStream != null) {
   				try {
   					outputStream.close();
   				} catch (IOException ignored) {
   				}
   			}
   		}
   	}
   }
   ```
   
   This snippet could help to investigate the performance differences in a scenario where a file is written to disk. On an old HDD I currently get these numbers:
   ```
   BufferedOutput took 1847ms
   serializeJavaObject took 2231ms
   serialize took 2756ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org


Re: [I] [Java] serializeJavaObject(...) is faster than serialize(...) [incubator-fury]

Posted by "chaokunyang (via GitHub)" <gi...@apache.org>.
chaokunyang commented on issue #1537:
URL: https://github.com/apache/incubator-fury/issues/1537#issuecomment-2074804065

   I run yoyr code locally, here is my result:
   ```java
   2024-04-24 08:12:19 INFO  Fury:144 [main] - Created new fury org.apache.fury.Fury@dfd3711
   BufferedOutput took 1275ms
   serializeJavaObject took 1975ms
   serialize took 1784ms
   ```
   serializeJavaObject/serialize should have similar result, there are not so much difference in those two methods


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org