; ; ; ; ; ; ; ; ; ; ; ; ; ;
I want to share with you a crash at customer site caused by java.lang.OutOfMemoryError.
Here is the original code:
if (synchRemove(lobj.getSeqNum()) != null);
____timeoutedList.add(lobj);
Can you see the problem?
(Well it's much easier after the relevant lines of code are isolated. In reality it took a few days and nights to get to these lines, remember it occurred in customer environment where not all relevant info is easily available for the development team. The OOM doesn't necessarily occur at this line).
Moral:
- Small semicolon can cause big troubles
- It's hard to see everything in code review. A trouble-making redundant semicolon can skip the eyes of the reviewer
- Load test may find such cases (but may still miss them, if the relevant scenario was not created)
- Good unit tests may also help
- Most Coding Guidelines require curly brackets for any block, even containing only one line. This could possibly reveal the error (if not by the developer, by the reviewer in code review)
- Static code analysis tools, like FindBugs – which is free, do point at such errors!
- In some IDEs (e.g. Eclipse) you can configure the IDE to present warnings on such cases
in the above case, analyzing the heap dump, using MAT, led to the problematic giant list, then tracking all insertions into the list in the code, led to the faulty line.
1 comment:
Hehe. Sorry. Super easy to miss! Didn't see it myself in the first couple of seconds.
That's why I'm never too lazy to add curly braces to an IF statement.
Also, on my team we have a FindBugs report as part of out nightly build+tests. This helps circumvents lazy programmers that never bother running FindBugs from the IDE.
We also add the below (IBM) JVM argument to auto generate a core dump in the event of an OOME. When analyzing core dumps with Eclipse MAT, you can see the actual in memory values, as appose to seeing the reference tree only (PHD file).
Then again these are all IBM terms, I might not anything new to Sun(ahhh... Oracle) JVM users. Donnno.
-Xdump:system+java+heap:events=systhrow+user,filter=java/lang/OutOfMemoryError,request=exclusive+prepwalk+compact
Post a Comment