Memory Leak in ObjectOutputStream

A memory leak exists in ObjectOutputStream that can cause your application to throw OutOfMemoryError.

 

To demonstrate this I have written a simple application that sends a lot (1500 per second) serialized small objects over the ObjectOutputStream via TCP connection.
During the testing I have discovered memory leaks and application crashed due to "out of memory".

First I have used a method ObjectOutputStream.writeObject(o).
This of course casued growing:

  private final HandleTable handles;
 private final ReplaceTable subs;
in ObjectOutputStream. This has prevented my objects to be garbage collected after sending and that was my intention.

My test does not need to keep cached objects so I have changed implementation to to ObjectOutputStream.writeUnshared(o).

My expectation was that this will end gowring tables in ObjectOutputStream.

Unfortunately  handles table have been still growing but my objects have been garbage collected. I have classified this as "memory leak".

Looking deeper into the problem the I have found possible reason.
The order of metthod calls is as follows:
ObjectOutputStream.writeUnshared(obj)
--- writeObject0(obj, true);
------ writeOrdinaryObject(obj, desc, unshared);
--------- handles.assign(unshared ? null : obj);
This last call is really:
    handles.assign(null);

Here you have a code for  ObjectOutputStream.HandleTable.assign(Object)
int assign(Object obj) {
    if (size >= next.length) {
growEntries();
    }
    if (size >= threshold) {
growSpine();
    }
    insert(obj, size);
    return size++;
}

What you can see here is that size of handle table is growing even if  this function is called with "null" argument. Increasing handles table is causing VM to run out of memory after some time.

 

WORKAROUND:
Call from time to time:
   ObjectOutputStream.reset()
This will clear handles.

Note: You can never be sure that you will reset it on time in live system.