Freaking behavior of a small little C/C++ bug

Oh boy.
Read till the end the event and its root cause. Important morals follow below.





We run systems that on high capacity events handle thousands of transactions per second. One of the most heavy-traffic periods is New-Year's-Eve, the 31st of December, were most of our systems are under heavy stress around the world, stress that tends to difuse to our support teams. Structured and strict preparations usually make us pass this heavy-traffic day properly in most, if not all sites. Which happily was the case also this year.

Shockingly, on January 2nd we had a crash in two sites.

Analyzing the crash led to a timer that instead of re-scheduling itself for every 5 seconds, keeps snapping abruptly in periods of milliseconds.

While still analyzing the case, reproducing it in our labs, the problem vanished as suddenly as it appeared, on the end of the same day. January 3rd, 00:00, systems went back to behave nicely.

That's really odd. How does the bug relates to the date? Is it a coincidence? It doesn't look so, as a second after midnight problem disappears. Trying to reproduce it in the lab we got the same behavior: it is the bug of January 2nd 2011. (By the way, when running the system in our labs in debug mode, problem didn't reproduce! Bug appears only when running without debug! That's common for memory related bugs, smears etc.)

To some of us, it sounded like the iPhone alarm bug. Which was reported also not to work properly on 2011 start, being fixed on its own, by January 3rd.

http://www.tipb.com/2010/12/31/iphone-bugs-alarms-working-2011/


Maybe it's the same bug?

iPhone runs on iOS which is Linux based. We also run on Linux. Maybe there is something with Linux timers on beginning of 2011?
Looking for something in this direction led to nothing.

On the other hand, analytical investigation led to the following:

1. The timer, when awakes, calls our callbak function. The callback function shall return an int value. Any value except 1 says "OK", 1 says - please call me again.
2. Our callback function didn't return a value at all

Wait... - is it legal not to return a value from a non-void method?
Unfortunately, in C/C++ it is. And the bevior is undefined. The function do return a value, in some environmnets it will be the last value from the register. And, well, occasionaly it can be 1.
See:
http://stackoverflow.com/questions/1610030/why-can-you-return-from-a-non-void-function-without-returning-a-value-without-pro/1610454#1610454
http://stackoverflow.com/questions/2598084/function-with-missing-return-value-behavior-at-runtime
What shall be done?
Read:
http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
-Wreturn-type
Warn whenever a function is defined with a return-type that defaults to int. Also warn about any return statement with no return-value in a function whose return-type is not void (falling off the end of the function body is considered returning without a value), and about a return statement with an expression in a function whose return-type is void. For C++, a function without return type always produces a diagnostic message, even when -Wno-return-type is specified. The only exceptions are `main' and functions defined in system headers. This warning is enabled by -Wall.

Morals

• Listen to compiler warnings!
  Solve all warnings, you should have a zero warnings policy.
  The problem above could be caught and solved as a warning (-Wreturn-type).
  
• If you don't keep a policy of zero warnings, which you should, turn bad warnings as the one above into an error, with a compilation flag, e.g.: -Werror=return-type
  
• You may want to test your software in future time, for example, have a test system that runs all the time 30 days ahead, if there is a time related bug it may help catching it on time. It won't probably catch everything, but it could have catch the problem we had above!

Semi-colon and java.lang.OutOfMemoryError

; ; ; ; ; ; ; ; ; ; ; ; ; ;

I want to share with you a crash at customer site caused by java.lang.OutOfMemoryError.

Here is the original code:

if (synchRemove(lobj.getSeqNum()) != null);
____timeoutedList.add(lobj);

Can you see the problem?

(Well it's much easier after the relevant lines of code are isolated. In reality it took a few days and nights to get to these lines, remember it occurred in customer environment where not all relevant info is easily available for the development team. The OOM doesn't necessarily occur at this line).


Moral:

1. Small semicolon can cause big troubles
2. It's hard to see everything in code review. A trouble-making redundant semicolon can skip the eyes of the reviewer
3. Load test may find such cases (but may still miss them, if the relevant scenario was not created)
4. Good unit tests may also help
5. Most Coding Guidelines require curly brackets for any block, even containing only one line. This could possibly reveal the error (if not by the developer, by the reviewer in code review)
6. Static code analysis tools, like FindBugs – which is free, do point at such errors!
7. In some IDEs (e.g. Eclipse) you can configure the IDE to present warnings on such cases
   

in the above case, analyzing the heap dump, using MAT, led to the problematic giant list, then tracking all insertions into the list in the code, led to the faulty line.

Java Unit Testing

There are a lot of tools and options out there for java unit testing, replacing JUnit or on-top of JUnit. Let's do some order in things.

1. TestNG vs. JUnit4

TestNG came out to add features that were missing in JUnit 3.x.
It did quite a good job and some may say that TestNG 5.10+ is still better than JUnit 4.7+
TestNG has better data providing mechanism.
TestNG has test dependency definition, missing in JUnit.
TestNG has group level with fixtures on group. And it also has the ability to create automatic run file for all failed tests.
All that said, JUnit has more extending libraries using its ability to extend JUnit TestCase and JUnit TestRunner.
Both have very good IDE support, as well as Ant and Maven support.

Coming to choose, both are good. I stick with JUnit.

Note that JUnit 4 made the mistake of integrating another library into its jar, that is the hamcrest core. They should have known better than that... The problem is that when you get the full hamcrest lib (to get the really powerful matchers that you need in your tests) it gets confused with the lib that came with JUnit. Solution is simple: put the full hamcrest that you bring to be first in the classpath, before the JUnit jar.


2. Load

JUnitPerf do the job with its ability to run tests in several threads.
It's easy and useful, for integration stage and for any piece of code that may act differently under load.

The following code was tested with JUnitPerf and it catches the bug of not synchronizing the addIfAbsent:

public class MyList extends ArrayList
// we test with the synchronized and without
public synchronized boolean addIfAbsent(Object o)
boolean absent = !super.contains(o);if(absent) {
super.add(o);
}
return absent;
}
}


The test that catches the bug when the synchronized is deleted:

public class MyListTest extends TestCase {
private MyList myList = new MyList();
private int count = 0;
MyListTest(String name) {
super(name);
}

TestSuite suite = new TestSuite();
Test testToRun = new MyListTest("testMyList");
int numUsers = 1000;
int iterations = 5;
suite.addTest(new LoadTest(testToRun, numUsers, iterations));
return suite;
}
public void testMyList() {
if(myList.addIfAbsent(count)) {
// there is a gap here that may cause error on correct
// implementation, in case of thread switch at this point
synchronized (this) {
++count;
assertEquals(myList.size(), count);
}
}
}
}

This solution is not bullet proof.
Theoretically it may give errors on correct scenario, and of course it can always miss bad implementation, as it’s a matter of timing.
However, in reality it does catch the problem!
JUnitPerf is simple and gets the job done.


3. Thread testing

It could be nice if we could have time the behavior of the code under test, to mimic race conditions and check how the code operates. The idea is to deliberately create race condition then to check that a synchronization lock is working as expected, ensure that unlocked blocks are fine, check for deadlocks and see if there data corruption.

With such ability we could have create the race condition in the above code with a deterministic approach instead of using load.

Suggested tools:

• thread-weaver (version 0.1)
  allows the test to time different threads reach code points in a certain order, either explicitly or “semi-automatically”
  
• MultiThreadedTC (version 1.01)
  allows to set time tickers on objects in the test itself, but NOT on the tested code, thus less relevant for most testing purposes

Both tools are not highly supported, recommending still not to rush into it…

4. Mock Objects

There are 3 cases where you need a Mock Object in your test:

(a) Tested code gets a complicated object as parameter
Solution: Mock the parameter with a Stub/Mock implementation

(b) Tested code invokes a static call on some resource
Solution: Replace the static call with a Stub/Mock implementation

(c) Tested code creates complicated objects
Solution: Replace the created objects with a Stub/Mock implementation

The Mock utilities rely on one of the following technologies: java.reflection.Proxy, replacing the ClassLoader, byte code instrumentation and AOP weaving (which relies by itself on replacing the ClassLoader or on byte code instrumentation).

Possible tools: Mockito, JMock, JMockit, EasyMock, JEasyTest and many others…

There are differences between the tools in syntax and abilities.
Tools that rely on Proxy cannot mock static behavior and object creation.

The two possibile combinations that came to my final round were:
(a) JMockit 0.998
(b) EasyMock 3.0 + PowerMock .1.3.8

JMockit seems powerful and well documented, but in one of my tests the test failed without a reason and when I debugged it I saw that some exception is thrown within the JMockit code itself. Probably I did something wrong in the test, but still this is not what I expect for. Maybe the 1.0 version would be better...

My selection here would be EasyMock + PowerMock


5. Other tools

Other tools in this domain worth mentioning:
- DBUnit
- HttpUnit (not only for tests)
- Selenium (it is a good tool for Web UI tests in general, and you can operate it from your java unit tests if you'd like)

Test Automation ROI

Recently I was working on a ROI (Return-on-Investment) Model for Test Automation. It was quite interesting to note that there are not too many "ready-made-excels" out there for this purpose. The ones that exist come mainly from automation tool manufectures. So I went to creating my own model, resulting with interesting notions.

Test Automation ROI Ingredients

Expenditure side

1. NRE (Non-Recurring Effort / Expense) - buying tool license, training people, raising the environemnt and infrastructure
2. Implementing the initial testing scenarios
3. Maintaining the testing scenarios and the environemnt and infrastructure
4. Adding new scenarios and new required features to the environemnt and infrastructure
5. Usually requires better trained testing engineers to maintain the automated testing system

Revenue side

1. In the long run, may be able to reduce manpower (less manual testing is needed)
2. Reduce number of escaped defects
3. Increase development confidence, allowing more features per release, thus less releases per year
4. Shorter test and development cycles and improved TTM (Time-to-Market)
5. Enhances the abilities of the testing team ("soft" revenue)

Influencing Factors

When coming to real life numbers it appears that it is very tricky to achieve positive ROI. And even if there is positive ROI the break even point comes after at least one year of investment, and in most cases two years. It's a simple case of big lump sum investment for future expected revenues.

The ROI is strongly related to the amount of bugs detected in the system in general and specifically in the field, number of existing manual testing engineers and the number of releases per year:

• A system with very low amount of bugs that are mostly found at testing room and not in the field, would make it harder for automation to have positive ROI, as the potential of reducing escaped defects is low
• If the current existing number of manual testing engineers is small, potential of revenues in reducing manpower is low. However, if number of escaped defects is high automation can assist in lowering it
• Low number of releases per year would also make it harder to achieve positive ROI, as the lump sum investment is made for less cycles of use per year

The ROI is strongly related to the changes in the system:

• Need for new scenarios and features in the environemt along the year means more investment in maintaing the testing system up-to-date
• The lifetime of an automated scenario before it becomes broken or irrelevant because of changes in the system is very important

On an "average" system with some major new features each year, about 4 major releases per year and about 20 major escaped defects per year, following can be found:

• In order to properly support and maintain a test automation project, you would probably need the same amount of people as you had before with manual testing, all along. The new features in your product requires new testing scenarios as well as new features in the test automation environment
• The major revenues stem from reducing number of major releases (adding more features per release) and from reduced number of defects in field
• It takes about 18 months to return the investment

Projects that are not stabilized yet and have too many changes would probably not earn from end-to-end test automation, although automated regression based on UnitTesting can still assist.