You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fury.apache.org by "istinnstudio (via GitHub)" <gi...@apache.org> on 2024/09/27 22:30:19 UTC
[I] A Class that implements Map interface cannot be deserialized, ... OutOfBoundsException [fury]
istinnstudio opened a new issue, #1863:
URL: https://github.com/apache/fury/issues/1863
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/fury/issues) and found no similar issues.
### Version
0.70.1
### Component(s)
Java
### Minimal reproduce step
There is a class that implements Map interface. This class cannot be deserialized. I will provide full reproducible example.. The example contains 2 classes, the customHashMap class and a deep copy example I use. Uncomment the field to switch between a valid Map and the erroneous explicit CustomHashMap Object
```
import java.io.ByteArrayOutputStream;
import java.nio.ByteBuffer;
import org.apache.fury.Fury;
import org.apache.fury.config.CompatibleMode;
import static org.apache.fury.config.Language.JAVA;
import org.apache.fury.logging.FuryLogger;
import org.apache.fury.memory.MemoryBuffer;
import org.apache.fury.memory.MemoryUtils;
public class DeepCopyApacheFury {
// Method to deep copy an object using Apache Fury with dynamic buffer allocation
public static Object deepCopyFuryWithByteBufferExample(Object theObj) throws Exception {
long startTime = System.nanoTime(); // Start time for performance tracking
// Initialize Apache Fury serialization
Fury fury = Fury.builder()
.withLanguage(JAVA) // Use Java language configuration
.withRefTracking(true) // Enable reference tracking (if needed)
.requireClassRegistration(true) // Ensure all classes are registered
.withCompatibleMode(CompatibleMode.SCHEMA_CONSISTENT)
// .withMetaShare(false)
// .withScopedMetaShare(false)
// .withRefCopy(false)
// .withAsyncCompilation(false)
.build();
// Manually register classes to improve performance
registerFuryLargeObjectExample(fury);
System.out.println("Starting Fury serialization...");
// Use ByteArrayOutputStream for dynamic buffer allocation
ByteArrayOutputStream byteArrayOutStream = new ByteArrayOutputStream();
Object newObj = null;
try {
// Serialize the object into the ByteArrayOutputStream
fury.serialize(byteArrayOutStream, theObj);
byte[] serializedData = byteArrayOutStream.toByteArray();
long serializationEndTime = System.nanoTime();
System.out.println("Serialization completed in " + (serializationEndTime - startTime) / 1_000_000_000.0 + " seconds");
// Calculate the size of the serialized data in MB
double serializedSizeMB = serializedData.length / (1024.0 * 1024.0);
System.out.println("Serialized object size: " + serializedSizeMB + " MB");
// Allocate a ByteBuffer of the exact size of the serialized data
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(serializedData.length);
byteBuffer.put(serializedData);
byteBuffer.flip(); // Switch to reading mode
// Wrap the ByteBuffer in a MemoryBuffer for deserialization
MemoryBuffer memoryBuffer = MemoryUtils.wrap(byteBuffer);
System.out.println("memoryBuffer size = "+memoryBuffer.size());
// Deserialize the object from the ByteBuffer using Fury
System.out.println("Starting Fury deserialization...");
newObj = fury.deserialize(memoryBuffer);
long deserializationEndTime = System.nanoTime();
System.out.println("Deserialization completed in " + (deserializationEndTime - startTime) / 1_000_000_000.0 + " seconds");
} catch (Exception e) { // not in use as it breaks fury debugging log
// getLogger(DeepCopyApacheFury.class.getName()).log(System.Logger.Level.DEBUG, "Exception in DeepCopyApacheFury", e);
FuryLogger furyLogger = new FuryLogger(Object.class);
String msg = "FURY LOGGER: Exception in DeepCopyApacheFury";
furyLogger.error(msg, e);
// getLogger(DeepCopyApacheFury.class.getName()).log(System.Logger.Level.DEBUG, "Exception in DeepCopyApacheFury", e);
}
// Verify types for debugging
System.out.println("------------------ VERIFY ------------------");
System.out.println("Original object type: " + theObj.getClass().getName());
if (newObj!=null) {
System.out.println("Deserialized object type: " + newObj.getClass().getName());
} else{System.err.println("newObj IS NULL");}
return newObj;
}
private static void registerFuryLargeObjectExample(Fury fury) {
fury.register(LargeMapObject.class);
fury.register(CustomHashMap.class);
}
// Main method demonstrating the deep copy with large object
public static void main(String[] args) throws Exception {
// Create a large object (e.g., Map with a large number of entries)
LargeMapObject originalObject = new LargeMapObject();
originalObject.populateLargeMap(1_000_000); // Populate with 1,000,000 entries
// Perform deep copy using Fury
LargeMapObject copiedObject = (LargeMapObject) deepCopyFuryWithByteBuffer(originalObject);
// Check if deep copy was successful (you can add further validation if needed)
System.out.println("Original object size: " + originalObject.getMapSize());
if (copiedObject!=null) {
System.out.println("Copied object size: " + copiedObject.getMapSize());
} else{System.err.println("COPY OF MAP IS NULL");}
}
// Example class to demonstrate serialization with large data
static class LargeMapObject {
// private Map<String, Integer> largeMap = new HashMap<>();
private CustomHashMap<String, Integer> largeMap = new CustomHashMap<>();
// Populate the map with a large number of entries
public void populateLargeMap(int numEntries) {
for (int i = 0; i < numEntries; i++) {
largeMap.put("key" + i, i);
}
}
// Get the size of the map (for validation)
public int getMapSize() {
return largeMap.size();
}
@Override
public String toString() {
return "LargeObject{mapSize=" + largeMap.size() + '}';
}}
}
```
```
import java.io.Serializable;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
@SuppressWarnings({ "rawtypes", "unchecked" })
public class CustomHashMap<K, V> implements Map<K, V> ,Serializable , Cloneable { // Cloneable for deep clone method
private Map<K, V> entryMap;
// SET: Adds the specified element to this set if it is not already present.
private Set<K> entrySet;
public CustomHashMap() {
super();
entryMap = new HashMap<K, V>(); // changed to linked
entrySet = Collections.newSetFromMap(new HashMap<>());
}
@Override
public Set<Map.Entry<K, V>> entrySet() {
return new HashSet<>(entryMap.entrySet());
}
@Override
public V put(K key, V value) {
V oldValue = entryMap.get(key);
insertionRule(key, value);
return oldValue;
}
@Override
public void putAll(Map<? extends K, ? extends V> t) {
for (Iterator i = t.keySet().iterator(); i.hasNext();) {
K key = (K) i.next();
insertionRule(key, t.get(key));
}
}
@Override
public void clear() {
entryMap.clear();
entrySet.clear();
}
@Override
public boolean containsKey(Object key) {
return entryMap.containsKey(key);
}
@Override
public boolean containsValue(Object value) {
return entryMap.containsValue(value);
}
public Set entrySetOriginal() {
return entryMap.entrySet();
}
@Override
public boolean equals(Object o) {
return entryMap.equals(o);
}
@Override
public V get(Object key) {
return entryMap.get(key);
}
@Override
public int hashCode() {
return entryMap.hashCode();
}
@Override
public boolean isEmpty() {
return entryMap.isEmpty();
}
@Override
public Set keySet() {
return entrySet;
}
@Override
public V remove(Object key) {
entrySet.remove(key);
return entryMap.remove(key);
}
@Override
public int size() {
return entryMap.size();
}
@Override
public Collection values() {
return entryMap.values();
}
// Method to return a Map from a CustomHashMap. It can return a map anyway but it is more obvious this way
public Map<K, V> getMap() {
return this;
}
public Map<K, V> getEntryMap() {
return entryMap;
}
public Set<K> getEntrySet() {
return entrySet;
}
@Override
public Map<K, V> clone() {
try {
CustomHashMap<K, V> clonedMap = (CustomHashMap<K, V>) super.clone();
clonedMap.entryMap = new HashMap<>(entryMap); // Shallow copy the entryMap
clonedMap.entrySet = new HashSet<>(entrySet); // Shallow copy the entrySet
return clonedMap;
} catch (CloneNotSupportedException e) {
// This should never happen since CustomHashMap is Cloneable
throw new InternalError(e);
}
}
public synchronized boolean insertionRule(K key, V value) {
// KEY as null and EMPTY String is not allowed.
if (key == null || (key instanceof String && ((String) key).trim().equals("") ) ) {
return false;
}
// If key already available then, we are not overriding its value.
if (entrySet.contains(key)) { // Then override its value, but we are not allowing
return false;
} else { // Add the entry
entrySet.add(key);
entryMap.put(key, value);
return true;
}
}
}
```
### What did you expect to see?
a deserialized object
### What did you see instead?
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: Range [260, 2) out of bounds for length 262. This probably is for the example, could be different on other Map content, generally is "OutOfBoundsException"
### Anything Else?
I have various error messages as this custom map class is being used on more complex classes, and I have a fair conclusion that this one creates several issues, but I am not entirely sure. This explicit customhashmap usage was done by mistake and stayed there for one small object even if it is a bit "heavier" comparing to the native Map as it is now an Object. Eventually it will be replaced by a true Map but that will be tricky as versioning will take place in various classes. FST serializer v2.57 works OK with it, so there should be a bug or a missing case in Fury...
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@fury.apache.org
For additional commands, e-mail: commits-help@fury.apache.org