On Software and Other Animals

Big Map, String, GC, Memory Consumption and Performance

2011-08-18T04:03:00.000-07:00

Hashtables are used in many cases to cache information in memory. There are other alternatives, like in memory databases, or flushing data to disk, but when all you need is a simple key-value relation and you need very high performance, the simple Java HashMap can do the trick. Usually cache can allow loosing information that was not claimed recently, assuming that recent usage of the data can foresee higher chances of this data to be claimed again soon, compared to less recently used data. But in some cases we prefer the cache to hold as much information as possible, which means a VERY BIG MAP.

Important morals below are based on an application with HashMap holding 20M entries, in 64bit JVM and 6 GB (-d64 -Xmx6g ). The Map itself uses much less memory, but in order for the entire application to work smoothly we need the 6 GB.

When coming to very big HashMap, there are a few considerations to take into account:

The default load factor of 0.75 is reasonable. Don't play with it without understanding what you are doing. The load factor determines when the map should grow and rehash should occur, this happens when number of elements in the map gets above the result of (capacity * load factor). Higher values of load factor means less rehash operations, but a map that is more dense. Calls to get in a dense map are more costly since buckets are full with many entries that shall be iterated. Lower load factor values will create more rehash operations and a sparse map, which is more memory consuming but more efficient in get calls. It should be noticed that load factor can be bigger than 1, which means we allow the table (number of buckets) to be smaller than number of entries.

To summarize: don't start with optimizing the load factor, it's a bad start.

Initial capacity is important. Starting the HashMap too small will result with rehash operations upon calls to put. Though HashMap doubles itself on resize, still failing to provide appropriate initial capacity may result with a few costly resize cycles.
The recommended initial capacity, for load factor of 0.75, is: 1.5 * numElements
(Read more: http://stackoverflow.com/questions/434989/hashmap-intialization-parameters-load-initialcapacity).

In our case, the performance difference of setting 20M entries into the map, with setting the initial capacity to 30M (Java will in fact set the initial capacity to the closest power of 2 above this size), compared to no initial capacity set, was almost double the insertion time, with very clear workload on rehash and GC work following it.

To summarize: initial capacity is important. Set it!

Beware of memory leaks!
Maps, as any long-leaving memory storage entity, are a potential source for leaks.
In our case, the Map got Strings from file, but needed to use for the entries' keys only a small portion of each line read from the file. Thus, after reading each line from the file, some string tokenizing was done resulting with the key and value to store.
Unfortunately the entire line read from file is kept in memory, even though we need only part of it... This is due to the way substring and other similar tokenizing methods are implemented, which return a new String, that holds the old one with offsets inside. This is a known issue (for example see: http://eyalsch.wordpress.com/2009/10/27/stringleaks/).
Solution is to manually create a new String and give it the substring portion you want to keep:
String key = new String(bigLine.substring(from, to));
In our case, fixing a similar issue resulted with requiring 1.5GB less memory for the exact same scenario! It turned out that the map keys' kept a long tail that should have been cut.

To summarize: beware of holding a substring in a Map! Do it the right way.

In such a huge map, every byte in the key-value becomes 20M in the entire map. Thus saving a few bytes can worth the hassle. For example, instead of using String as the value, we decided to use char array (a simple char[]) thus saving: [1] a reference to the string object from the hashmap entry - instead holding a direct reference to char[]. In 64 bit this values to 8 bytes saving! [2] three int fields that are in String class: offset, count and hashmap. The offset and count fields are there to allow String to point at a position which is partial in the char[] it holds (directly related to the substring leak mention in item 3 above). We don't need this info. Also caching the hashmap value is redundant, as the Map entry itself does it.

By using char[] instead of String as our value, we save 20 bytes per entry, which is 400MB total! And without hurting the code or making it more complex.

Our key is also a String, but it cannot be turned into char[], as we need a proper hashcode and equals functions. Still, one should consider implementing a dedicated Key class that holds only the relevant info, char[] in our case. Since the extra reference cannot be saved, the saving here could be the 3 int fields in class String, which totals to 240MB for 20M entries.

To summarize: caching Strings in big Maps is costly. Think how to reduce the size of your key and value!

The hashcode is crucial. Bad hashcode on the key, which creates bad distribution (too many keys get the same hashcode value) can be performance devastating. On the other hand, though of much less importance, a more efficient hashcode method has an influence over at least 20M insertion operations. Read how String hashcode is implemented to get the idea: http://stackoverflow.com/questions/299304/why-does-javas-hashcode-in-string-use-31-as-a-multiplier

To summarize: check your key hashcode method, use some real data and validate that you get reasonable distribution. If needed, work on improving the hashcode method by using higher power multiplication on the key fields.

Know when to stop optimizing. In some cases you see a programmer keep optimizing when there is really nothing more to optimize (well, there is always more to optimize, but then it may result with taking parts to C and using JNI, or use direct byte arrays, things you would better not go to, if not really needed...).
Calculate the most minimal amount of memory you need, save your time and don't try to optimize your application into less.

To summarize: set goals to your optimization efforts. Make sure the goals are not too aggressive, i.e. are feasible without re-writing the entire thing.

The entries of the big map will eventually arrive to the GC old-generation section. It means that in the described usage the old generation is going to be very big. There are two options to handle this, one is to play with the generation ratios, the other option is to set a big enough total heap memory to allow the old generation to be big enough. If you have enough resources on the machine you can go with the second option, otherwise you would need to dig into the generation ratio configuration.

To summarize: size of the old generation section is crucial to prevent unnecessary full GC cycles. Configure the max heap size and/or the generation ratios.

By optimizing the memory consumtion we got also much better performance, as the application went into less GC cycles. Some other parts of the drill include removing unused entries from the cache, in a dedicated thread, when reaching certain threshold, and more... The result is a huge cache running smoothly on a reasonable sized server.

Decent Code

2011-04-22T08:29:00.000-07:00

Retrospecting the posts labled Decent Code I see many good points worth memorizing.

Do you have an API? - the basic question before coding!

Do you have a theory? - the basic question before debugging!

Be proud of your code! - a simple advice, yet powerful, for code reviews

The risks of redundant code (or - Less is More) - another powerful advice for code reviews

Use Explaining Variables! - and another one for code reviews (I should consider a label for code reviews probably)

Resource Files - this is obvious for anybody today, yet ignored too often...

Freaking behavior of a small little C/C++ bug - avoid the non-void not returning a value!

Semi-colon and java.lang.OutOfMemoryError - the methodic way of analyzing OOM in Java programs

and many more...

like for example...

Make sure to have a strict XSD!

or To AJAX or NOT to AJAX? - important to many web developers today which neglect the request-response model for AJAX idol, in many cases for no good reason.

and still, many more...

The success of iPhone

2011-03-31T22:55:00.000-07:00

Three years back, I explained why Nokia is not supporting external applications.

Well, since then Apple has AppStore and Nokia has followed with OVI. And Samsung has its Internet@TV portal.

It appears that focusing on the appliance itself is not enough nowadays.

Thoughts following 2010 FogBugz and Kiln World Tour

2011-01-20T05:47:00.000-08:00

I was attending yesterday Joel Spolsky world tour for FogBugz and Kiln.

I must admit that till yesterday I thought that Joel has a great blog about SW development, but I didn't quite understand - who needs yet another bug tracking tool... is there still a real market for that, when you have today so many good free tools!?

A few thoughts following the event:

Joel's presentation skills do not fall from his writing. The gathering starting with a projected countdown analog clock, Joel went on stage right at the moment when the countdown reached its end - with a pre-planned bug on the clock finish resulting with some fake errors and blue screens, which led Joel to open a bug, start his presentation, then get a response on his newly opened bug "poping accidentally" while in his presentation (interestingly enough, while his Outlook is closed :-) , which led to getting into the code itself using Kiln, comparing versions and eventually "solving" the bug and checking in. A full bug detection and solving cycle in 10 minutes. With a few jokes here and there, and while going quickly through the products' abilities and strength points.

The traditional tools for bug tracking and source control are OK. But even in this "already solved" domain, there is still room for improvements and for new players, either open source or commercial. When we have an idea to develop something we usually tend to check who already done that and how good it is. And when we see that there are already several reasonable solutions we assume that the market is closed for us on that. Well, Joel shows that there is always a market for realy good products, you don't have to invent a new big thing, you may need however to invent many small things.

Distributed Source Version Control - GIT, Mercurial, Kiln - disconnects the developers own copy from a repository, while preserving full history and repository context. This concept has several significant advantages:
(a) You do not postpone your check-ins, being afraid of hurting the main repository, check-ins are made into your private copy. When you are ready you push your copy back to the main repository. Your check-ins preserve full history during your development! This is also important when you perform a merge and realize that your original file was overriden by a wrong change - you don't have to worry as you have your full history at hand and you can get back to your last check-in.
(b) You can push your developments to another repository, e.g. to a QA repository, thus allowing to push relevant fixes quickly, without releasing them yet to the development repository.
(c) Upon merge, it's easier to see for each change the exact origin of it. Managing several customer releases you can see which changes in the main release where merged into the customer release and which not.

Online IDE for almost any SW lang you can think of

2011-01-13T22:52:00.000-08:00

Take a look at this one: http://ideone.com/

It supports:
Ada, Assembler, AWK, Bash, bc, Brainf**k, C, C#, C++ se, C++0x, C99 strict, CLIPS, Clojure, COBOL, COBOL 85, Common Lisp (clisp), D (dmd), Erlang, F#, Factor, Falcon, Forth, Fortran, Go, Groovy, Haskell, Icon, Intercal, Java, JavaScript (rhino), JavaScript (spidermonkey), Lua, Nemerle, Nice, Nimrod, Objective-C, Ocaml, Oz, Pascal (fpc), Pascal (gpc), Perl, Perl 6, PHP, Pike, Prolog (gnu), Prolog (swi), Python, Python 3, R, Ruby, Scala, Scheme (guile), Smalltalk, SQL, Tcl, Text, Unlambda, Visual Basic .NET, Whitespace

It's very useful when you want to check little programs, like the ones I tried when writing Freaking behavior of a small little C/C++ bug:

[1]

[2]

Freaking behavior of a small little C/C++ bug

2011-01-13T22:02:00.000-08:00

Oh boy.
Read till the end the event and its root cause. Important morals follow below.

We run systems that on high capacity events handle thousands of transactions per second. One of the most heavy-traffic periods is New-Year's-Eve, the 31st of December, were most of our systems are under heavy stress around the world, stress that tends to difuse to our support teams. Structured and strict preparations usually make us pass this heavy-traffic day properly in most, if not all sites. Which happily was the case also this year.

Shockingly, on January 2nd we had a crash in two sites.

Analyzing the crash led to a timer that instead of re-scheduling itself for every 5 seconds, keeps snapping abruptly in periods of milliseconds.

While still analyzing the case, reproducing it in our labs, the problem vanished as suddenly as it appeared, on the end of the same day. January 3rd, 00:00, systems went back to behave nicely.

That's really odd. How does the bug relates to the date? Is it a coincidence? It doesn't look so, as a second after midnight problem disappears. Trying to reproduce it in the lab we got the same behavior: it is the bug of January 2nd 2011. (By the way, when running the system in our labs in debug mode, problem didn't reproduce! Bug appears only when running without debug! That's common for memory related bugs, smears etc.)

To some of us, it sounded like the iPhone alarm bug. Which was reported also not to work properly on 2011 start, being fixed on its own, by January 3rd.

http://www.tipb.com/2010/12/31/iphone-bugs-alarms-working-2011/

Maybe it's the same bug?

iPhone runs on iOS which is Linux based. We also run on Linux. Maybe there is something with Linux timers on beginning of 2011?
Looking for something in this direction led to nothing.

On the other hand, analytical investigation led to the following:

The timer, when awakes, calls our callbak function. The callback function shall return an int value. Any value except 1 says "OK", 1 says - please call me again.
Our callback function didn't return a value at all

Wait... - is it legal not to return a value from a non-void method?
Unfortunately, in C/C++ it is. And the bevior is undefined. The function do return a value, in some environmnets it will be the last value from the register. And, well, occasionaly it can be 1.

See:
http://stackoverflow.com/questions/1610030/why-can-you-return-from-a-non-void-function-without-returning-a-value-without-pro/1610454#1610454

http://stackoverflow.com/questions/2598084/function-with-missing-return-value-behavior-at-runtime

What shall be done?

Read:

http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html

-Wreturn-type

Warn whenever a function is defined with a return-type that defaults to int. Also warn about any return statement with no return-value in a function whose return-type is not void (falling off the end of the function body is considered returning without a value), and about a return statement with an expression in a function whose return-type is void. For C++, a function without return type always produces a diagnostic message, even when -Wno-return-type is specified. The only exceptions are `main' and functions defined in system headers. This warning is enabled by -Wall.

Listen to compiler warnings!
Solve all warnings, you should have a zero warnings policy.
The problem above could be caught and solved as a warning (-Wreturn-type).
If you don't keep a policy of zero warnings, which you should, turn bad warnings as the one above into an error, with a compilation flag, e.g.: -Werror=return-type
You may want to test your software in future time, for example, have a test system that runs all the time 30 days ahead, if there is a time related bug it may help catching it on time. It won't probably catch everything, but it could have catch the problem we had above!

A simple generic template function

2010-11-25T05:39:00.000-08:00

A simple generic template function for getting the minimum and maximum from STL container and simple arrays.

Nothing too complicated, I just liked this example.

1:  template<class Iterator>
2:  pair<Iterator, Iterator> minMaxFinder(Iterator begin, Iterator end)
3:  {
4:      Iterator min = begin;
5:      Iterator max = begin;
6:      for(++begin ; begin != end; ++begin) {
7:          if(*begin > *max) {
8:              max = begin;
9:          }
10:         else if(*begin < *min) {
11:             min = begin;
12:         }
13:      }
14:      return pair<Iterator, Iterator>(min, max);
15:  }
16:  int main()
17:  {
18:      int iArr[] = {15, 5, 70, 2};
19:      pair<int*, int*> minmax = minMaxFinder(&iArr[0], &iArr[4]);
20:      cout << *minmax.first << ", " << *minmax.second << endl;
21:      list<string> sList;
22:      sList.insert(sList.end(), "small");
23:      sList.insert(sList.end(), "smallish");
24:      sList.insert(sList.end(), "big");
25:      sList.insert(sList.end(), "biggish");
26:      pair<list<string>::iterator, list<string>::iterator> 
27:               minmaxS = minMaxFinder(sList.begin(), sList.end());
28:      cout << *minmaxS.first << ", " << *maxminS.second << endl;
29:      return 0;
30:  }

Code formatted with http://codeformatter.blogspot.com

Thoughts on Open Source licenses, Patents on Software and such

2010-11-22T07:16:00.000-08:00

I'm dealing with Open Source usage approval cycle, which is an important task in any company, last thing that you want is to have your developers use whatever they find on the web.

Few insights and thoughts from my experience:

Developers in genral are mostly ignorant to legal issues. If not controlled they may use a free 30-days evaluation copy embedded in their system, just because the word free appeared somewhere in the site. In most cases they don't bother to read the license.
With Off-Shore developers problem is even bigger. They tend to be much more open with open source, without seeing the risks. Even if they do follow the company policy, submitting usage requests for open source usage, you may find inside their code much more "embedded" un-approved snippets and libraries. I tend to think that the reason is the distance, they believe that even if caught the maximum you could do is yell at them over the phone or in e-mails, but you cannot beat them physically and they use it.
For above reasons and others, usage of open source must be controlled. There are scanning tools in the market that help you find un-reported usage of open source and commercial external software. Usage of such is helpful in finding the disobedient developers who still drop in whatever they like, fix that on time and beat them while the felony is still hot.
Scanning tools also point at many usages that are a very small snippet of something that looks like might be taken from an open source or even from an un-licensed example on the web. To some it seem a problem that should be fixed in the code, I personally believe that the rights on how to perform quick sort do not belong to anybody, even if part of some open source or are published on the web somewhere. Taking two notes from a melody doesn't harm its rights.
Same goes for patents on software. Publishing a patent on algorithm is problematic, but many patents are on "a method and a system". I have such one myself. Does it really prevent anyone from creating a new similar development? Should it?

Resource Files

2010-09-20T00:26:00.000-07:00

Programs interact with a user in many ways.
They write things to a screen, send messages over the network, say things aloud. And more.

In most cases, except maybe for debug logs, the string conveyed to the user shall be edited, and in some cases maybe even translated to other languages.

It's pitty to still see today modules that interact with a user, without using external resource files. Guys - how do you want someone to edit your lovely messages into something readable? and translate it into Swedish? or Yiddish?

The technical way of how to use resource file is a very old trick. The problem is that in the early days of a project, this is not the top prioirity (that's wrong, guys! the price at the beginning is very low!). The problem is that when it does become relevant, the code is full of strings in so many different places that it's a nightmare to do something.

Things to do on the first day when starting a new project (don't postpone it to "later"):
1. Use good logging mechanism (exiting one, don't invent the wheel)
2. Use automatic build mechanism
3. Use a resource file
( -- do you have more - please add as a comment!)

Well, wait I have one more - have a defined API (slash protocol slash wireframes) for the modules you are developing.

Semi-colon and java.lang.OutOfMemoryError

2010-08-19T05:24:00.000-07:00

; ; ; ; ; ; ; ; ; ; ; ; ; ;

I want to share with you a crash at customer site caused by java.lang.OutOfMemoryError.

Here is the original code:

if (synchRemove(lobj.getSeqNum()) != null);
____timeoutedList.add(lobj);

Can you see the problem?

(Well it's much easier after the relevant lines of code are isolated. In reality it took a few days and nights to get to these lines, remember it occurred in customer environment where not all relevant info is easily available for the development team. The OOM doesn't necessarily occur at this line).

Moral:

Small semicolon can cause big troubles
It's hard to see everything in code review. A trouble-making redundant semicolon can skip the eyes of the reviewer
Load test may find such cases (but may still miss them, if the relevant scenario was not created)
Good unit tests may also help
Most Coding Guidelines require curly brackets for any block, even containing only one line. This could possibly reveal the error (if not by the developer, by the reviewer in code review)
Static code analysis tools, like FindBugs – which is free, do point at such errors!
In some IDEs (e.g. Eclipse) you can configure the IDE to present warnings on such cases

in the above case, analyzing the heap dump, using MAT, led to the problematic giant list, then tracking all insertions into the list in the code, led to the faulty line.

... and the UI is also going back

2010-06-03T04:10:00.000-07:00

Following my last post on concurrent processes, my good friend Effie Nadiv noted that the UI is also getting backwards.

Back in the old days, Lotus 1-2-3 had a menu line that was not opening to overlay your main screen but was instead changing the menu line each time you make a menu selection. One can argue that this is not optimal, but I remember it as something very useful. Today in some cases you find yourself struggling to open the sub-sub-sub-menu to select something, trying not to close the entire thing before you make your selection.

It becomes also a common practice that any new dialog or window may hide previous ones, extra info may hide the main data etc. Effie argues that it all becomes from the philosophy that the application may do whatever it wants. Restrictions would heart creativity and we don't want that. So you have creativity in the new versions: my new version of Babylon has a fancy UI, ignoring the fact that it is not working, and the old one with the standard Windows UI was OK, it's a real charm. The new office is a nightmare, after you got used to something. Any new version of a SW tries to justify itself with a new fancy visual design which you don't want. You just want it to work.

Process Concurrency - are we stepping ahead or backward?

2010-06-03T02:25:00.000-07:00

The first version of Apple's iPad allows only one running process. A bit limited, but most people reported high satisfaction. On the other hand, my OS allows for multi processes to run in parallel and it starts to turn me crazy.

Why process concurrency is a BAD THING

You are showing a presentation and suddenly some pop-up from another process pops
Doing a critical debugging task, you are all focused and sharpened, but then your machine starts being sluggish as some background process decides to take CPU time or memory
You open your task manager to understand why your machine is so sluggish and see a list of so many processes. What's all that? I don't need half of it. Though, the first "terminate" that you try shuts down the system.
Any utility today can decide to run in your background, to really control what runs in your background you need to be an expert, otherwise you lose your background to all sort of things you really don't know much about.

How often do you click Ctrl+ALT+Del?

I find myself at least once a day using the Ctrl+ALT+Del to check what's holding my machine. How about you?

So, how was it back then, in the good old times?

In DOS, you could have only one process running. If you wanted to jump to another without closing your application you could use TSR but you could have only one TSR pop at a certain time, and when it was at the background it took NO resources, it just set there waiting for an interrupt without making any sigh.

The un-responsiveness that we get today from our systems is unbearable. Applications decide to go at the background to update themselves, or to index the file system, or to do other non-critical task. The feeling is that the resources of the machine are not yours. You are lucky if the machine throws a bone at you giving you some resources.

The user experience you get today on a quad core, xGhz CPU with xGB fast RAM are worse than what you got on an old old x8086 machine, 1000 times weaker. The difference is that the old applications were much more focused on being lean and mean, never went to the net during their operation (the web was BBS and gopher), and they didn't run in parallel.

We need to go back to the old approach, giving the stage to one application at a time - all other would be HALTED and would not take any system resources. It sounds so limiting, but this is how DOS used to work. And this is how the first version of iPad works.

Java Unit Testing

2010-05-31T07:45:00.000-07:00

There are a lot of tools and options out there for java unit testing, replacing JUnit or on-top of JUnit. Let's do some order in things.

1. TestNG vs. JUnit4

TestNG came out to add features that were missing in JUnit 3.x.
It did quite a good job and some may say that TestNG 5.10+ is still better than JUnit 4.7+
TestNG has better data providing mechanism.
TestNG has test dependency definition, missing in JUnit.
TestNG has group level with fixtures on group. And it also has the ability to create automatic run file for all failed tests.
All that said, JUnit has more extending libraries using its ability to extend JUnit TestCase and JUnit TestRunner.
Both have very good IDE support, as well as Ant and Maven support.

Coming to choose, both are good. I stick with JUnit.

Note that JUnit 4 made the mistake of integrating another library into its jar, that is the hamcrest core. They should have known better than that... The problem is that when you get the full hamcrest lib (to get the really powerful matchers that you need in your tests) it gets confused with the lib that came with JUnit. Solution is simple: put the full hamcrest that you bring to be first in the classpath, before the JUnit jar.

2. Load

JUnitPerf do the job with its ability to run tests in several threads.
It's easy and useful, for integration stage and for any piece of code that may act differently under load.

The following code was tested with JUnitPerf and it catches the bug of not synchronizing the addIfAbsent:

public class MyList extends ArrayList
// we test with the synchronized and without
public synchronized boolean addIfAbsent(Object o)
boolean absent = !super.contains(o);if(absent) {
super.add(o);
}
return absent;
}
}

The test that catches the bug when the synchronized is deleted:

public class MyListTest extends TestCase {
private MyList myList = new MyList();
private int count = 0;
MyListTest(String name) {
super(name);
}

TestSuite suite = new TestSuite();
Test testToRun = new MyListTest("testMyList");
int numUsers = 1000;
int iterations = 5;
suite.addTest(new LoadTest(testToRun, numUsers, iterations));
return suite;
}
public void testMyList() {
if(myList.addIfAbsent(count)) {
// there is a gap here that may cause error on correct
// implementation, in case of thread switch at this point
synchronized (this) {
++count;
assertEquals(myList.size(), count);
}
}
}
}

This solution is not bullet proof.
Theoretically it may give errors on correct scenario, and of course it can always miss bad implementation, as it’s a matter of timing.
However, in reality it does catch the problem!
JUnitPerf is simple and gets the job done.

3. Thread testing

It could be nice if we could have time the behavior of the code under test, to mimic race conditions and check how the code operates. The idea is to deliberately create race condition then to check that a synchronization lock is working as expected, ensure that unlocked blocks are fine, check for deadlocks and see if there data corruption.

With such ability we could have create the race condition in the above code with a deterministic approach instead of using load.

Suggested tools:

thread-weaver (version 0.1)
allows the test to time different threads reach code points in a certain order, either explicitly or “semi-automatically”
MultiThreadedTC (version 1.01)
allows to set time tickers on objects in the test itself, but NOT on the tested code, thus less relevant for most testing purposes

Both tools are not highly supported, recommending still not to rush into it…

4. Mock Objects

There are 3 cases where you need a Mock Object in your test:

(a) Tested code gets a complicated object as parameter
Solution: Mock the parameter with a Stub/Mock implementation

(b) Tested code invokes a static call on some resource
Solution: Replace the static call with a Stub/Mock implementation

(c) Tested code creates complicated objects
Solution: Replace the created objects with a Stub/Mock implementation

The Mock utilities rely on one of the following technologies: java.reflection.Proxy, replacing the ClassLoader, byte code instrumentation and AOP weaving (which relies by itself on replacing the ClassLoader or on byte code instrumentation).

Possible tools: Mockito, JMock, JMockit, EasyMock, JEasyTest and many others…

There are differences between the tools in syntax and abilities.
Tools that rely on Proxy cannot mock static behavior and object creation.

The two possibile combinations that came to my final round were:
(a) JMockit 0.998
(b) EasyMock 3.0 + PowerMock .1.3.8

JMockit seems powerful and well documented, but in one of my tests the test failed without a reason and when I debugged it I saw that some exception is thrown within the JMockit code itself. Probably I did something wrong in the test, but still this is not what I expect for. Maybe the 1.0 version would be better...

My selection here would be EasyMock + PowerMock

5. Other tools

Other tools in this domain worth mentioning:
- DBUnit
- HttpUnit (not only for tests)
- Selenium (it is a good tool for Web UI tests in general, and you can operate it from your java unit tests if you'd like)

Beware of your long tail garbage

2010-04-14T04:58:00.000-07:00

IBM’s PLDE seminar 2010 (IBM Programming Languages and Development Environments Seminar 2010)

An interesting session by Kathy Barabash on GC running with parallel cores raises an important note I haven't thought about: long referenced tails are harder for the GC to parallelize, as the GC cannot break long list efficiently into two threads. Thus if you create references with long distance from the root (local or static reference), your GC time would be longer compared to same number of references in a more spread structure.
XALAN seem to sin with the above.

Gilad Bracha opened with a overview of mistakes done by Java 1.0 which are living in Java till these days.

Gili Nachum writes on the above two in his JavaTuning blog:
http://www.javatuning.com/ibms-plde-seminar-2010-review/

Cisco new router deliverring 322 Tbit/s

2010-03-10T20:47:00.000-08:00

Cisco announced a new core router, named CSR-3, delivering 322 Tbit/s:
http://www.lightreading.com/document.asp?doc_id=188914&

My brother-in-law is part of this project. I'm not sure whether his part is the extra 22 Tera bits beyond 300, but he is there inside for the past years and has a major role there.

Whether this is a world shocking event for the internet world or not we will see, but for sure this is a landmark, comparing to the 56K b/sec not long ago (well, this is not an honest comparison, comparing home pace to core, but still the rate at the core level got from gigs to teras - factor of 1,000).

It's a very happy declaration for companies doing video services, IPTV etc. Of course there is still the need to propagate these paces to the homes, with fiber to the home, but this is already happening at some countries and will expand (Hong Kong, Hague, more Europe, and it happens that Kansas city even changed its name to get fiber).

With 322 Tbits/s it sounds that someone needs to really start working on teleporting.

Multiple Inheritance - should you?

2010-02-16T08:26:00.000-08:00

It is often argued that multiple inheritance in C++ is not a good practice. This is why Java doesn't have this ability. Of course, multiple inheritance in C++ for something that is similar to interface implementation as in Java is reasonable, this is the case when all classes inherited from are pure abstract with no data members and no implementaions, only pure virtual methods, except one base which is the true parent.

However, in some cases you do see in C++ multiple inheritance that is not just interface implementation. And in some cases it seems reasonable. The iostream library is using it quite a lot.

I had a code review in which the developer used multiple inheritance. He said he was considering whether to use it or not and decided it serves his goal. Indeed it was OK, there was a class "SomeSortOfEventHandler" that was of both types "A_CertainEventHandler" and "AnotherKindOfEventHandler", that is, the "SomeSortOfEventHandler" was in fact handler for two kinds of events. In this case there was no need for virtual inheritance from the top parent ("AbstractEventHandler"), as the special "SomeSortOfEventHandler" need to have the data and behavior of both its parents.

So far so good. But at a certain point there was a need to implement a virtual method in two flavors, one for being the child of one parent and the other for being the child of the other. This becomes nasty. One can add to the virtual method a parameter of type T (there is a template class at the top, that differentiate the different families under "AbstractEventHandler") - but this is not so elegant as there is no real need for this parameter. All other alternatives were also breaking the elegancy of the inheritance tree. Conclusion: better to create two separate classes and hold a pointer from one to the other to get the relation between two objects, rather than use multiple inheritance.

Array of Generic Type

2009-11-22T06:48:00.000-08:00

I was asked why one cannot create an array of generic type, which means this line cannot compile:

// the below doesn't compile
ArrayList<Integer>[] arrayOfLists = new ArrayList<Integer>[7];

The compiler doesn't like the creation part, on the right, while the declaration alone, on the left, goes fine. This issue is widly discussed, e.g. nicely described here.

There are many ideas why it is forbidden and many suggestions for workarounds. But the nice colleague who asked me this question liked the most the workaround I suggested him, which is very simple.

I started by asking why he needs this.

And he explained that he needs to manage some info for 7 days a week:

// the below doesn't compile
ArrayList<SomeInfo>[] weeklyInfo = new ArrayList<SomeInfo>[7];

I suggested to create two new classes, DailyInfo and WeeklyInfo:

class DailyInfo {
    private ArrayList<SomeInfo> dailyInfo = new ArrayList<SomeInfo>();
    ...
}

class WeeklyInfo {
    private DailyInfo[] dailyInfo = new DailyInfo[7];
    ...
}

In most cases this would be the right object oriented approach, which by the way solves the problem of array of generic type.

m and n have switched on my keyboard...

2009-10-07T23:02:00.000-07:00

The letters 'm' and 'n' have been switched. I type 'm' and it comes out as 'n'. I remember something about a virus so I run my anti-virus (no result) and search the web (some results).

Cone om! I’n beconimg imsame!

It's so annoying. Maybe someone is fooling around with me.

A quick glance at another keyboard: it's not that 'm' types 'n' -- the keys themselves changed places... each sits at the other one's location. A quick pull of the keys, stucking them again back into their right position and everything is great!

After a short investigation it appears that my 6 years old son, who is using my computer for playing, accidentally popped out the 'm' and 'n', then put them again in place quickly before I'll notice. Well, almost in place... He didn't intentinally planned for giving me a hoax, though it came out quite a good one for his age.

Auto-unboxing hazard -- the three states of Boolean: TRUE, FALSE, and...

2009-09-23T06:23:00.000-07:00

What do you say about the following code:

boolean isActive = (Boolean) pageContext.getAttribute(IS_ACTIVE);
if(isActive) {
// dosomething
}

If you are worried about the casting to Boolean in case where attribute IS_ACTIVE doesn't exist, don't worry, it's OK to cast null to any type, null stays null.

So, is it OK?

Well, no. The problem is with the auto-unboxing to primitive type boolean, which throws NullPointerException if the Boolean is null. Which is a bit hidden from the programmer who might miss this hazard when writing the code.

How should we fix it?

Is the following a decent fix:

Boolean isActive = (Boolean) pageContext.getAttribute(IS_ACTIVE);
if(isActive) {
// dosomething
}

If you ask "what's the fix?" - we turned the variable isActive to be Boolean rather than boolean. Which should eliminate any NullPointerException on the first line! But only in order to pass the exact same NullPointerException into the 'if'...

So, what about:

Boolean isActive = (Boolean) pageContext.getAttribute(IS_ACTIVE);
if(isActive.booleanValue()) {
// dosomething
}

Yes, you are right, same NullPointerException. Maybe now even more explicit.

We could go with:

Boolean isActive = (Boolean) pageContext.getAttribute(IS_ACTIVE);
if(isActive != null && isActive) {
// dosomething
}

But I believe this looks better:

Boolean isActive = (Boolean) pageContext.getAttribute(IS_ACTIVE);
if(Boolean.TRUE.equals(isActive)) {
// dosomething
}

Frameworks -- Explicit and Implicit Wiring

2009-09-22T23:05:00.000-07:00

In any new project, there is that moment when you are reflecting whether a group of classes belong to the package "core" or "utils" or "framework", or whether you need all these three altogether.

The difference between utils, framework and core should be simple. Utils are the basic functions needed by everyone, mostly unrelated to this specific project. They are all stateless by definition, simple input and output, with no dependency with the project's structures. Utils can always be easily tested as stand alone. Core is the basic stuff. The stuff you would direct a new developer to learn first. It is project specific. It may include the basic types relevant for this project, interfaces, enums, project's basic exceptions, etc. Make sure to avoid making it a trash can. It's not the address for things without a context.

Framework is the tricky one of all three. As opposed to "utils", the role of "framework" as suggested by its name is more complex. A framework is a piece of behavior interacting with other code to represent a real, concrete, runtime behavior. Usually, even if built specifically for this project, it has some generic characteristics, it is written in a way that could be extended, or used in the future in other ways that we don't see currently. For this we use design patterns like strategy, command, visitor, and in fact almost any that you can think of, to create some kind of "dynamic behavior", or "dynamic wiring". Which means that the framework describes a behavior, but not entirely, only an exact usage can describe an end-to-end scenario. This is why frameworks are best explained with an example (while utils can be explained by what they simply do).

There are good frameworks and bad ones. EJB in its first versions for example was horrible (some don't like it till today, a first bad impression is hard to fix). JDBC is a good framework. Java-Servlets is a good framework. What makes good frameworks good is that people find them easy to use, they save development time copmpared to other alternatives, and you would usually find over time very few versions of the same good framework (and the new versions would be fully compatible with old code). Of course the question is how to achieve this simplicity, real contribution and long-run future compatibility.

When designing and implementing a framework, occasionally you want to make life easier for the user by doing something automatic for him, let's say pass the value of some field into another field with a relevant name convention -- thus the user doesn't have to write a single line of code! There is this trick in the framework which does it for him. Let's call it "implicit wiring". Alternatively, you may expose to the user a way to acquire the value and require him to pass it explicitly by writing his own code. Let's call it "explicit wiring".

In some cases the difference is not so obvious. Suppose you have a controller which may implement many kinds of behavior, if not implemented than a default behavior is used by the framework. When the number of different behaviors of this controller start to inflate, things become a bit obscure, unreadable, thus maybe "implicit". A better approach would be to break the controller to different interfaces and allow the user to implement the interfaces representing the relevant behavior for him, thus checked by the compiler that no method is accidentally left out, more readable and "explicit".

This discussion came out recently with my team member Effie. We had a reference between two UI elements, "table" and "toolbar". I suggested that if "table" doesn't declare on its referenced counterpart "toolbar" than we will assume by default that it relates to a toolbar with some predefined name (e.g. "toolbar"). That way, I said, even if the user forgot to give a name for his toolbar, things would work nicely.

Effie's reaction was a bit frightening. (Well, not for someone who knows him). It is common in our work environment that a worker shouts at his boss. Which happened in this case. "I'm not working this way, if you want such a framework leave me out of it", was the part I can quote. Of course when he chilled out came the explanation. Implicit wiring is the base of all framework sins. I like such semi-emotional semi-professional reactions! And he was totally right. Think of someone trying to understand how the framework connects a toolbar to a table. And if he wants to add now another table and toolbar how can he do it? True, there is still the explicit way of referencing by name, but once I entered the implicit way, most cases would be using the implicit format which makes the explicit less known and less documented in examples. Also, implicit wiring tends to break over time, as at some point it may collide with something else and break old code. On the other hand, if you allow only the explicit way, all examples would present the explicit way, code would be always more simple to understand and maintain, and less prone for future incompatibilities. Even with the costs of higher efforts in the first implementation, explicit shall win.

Frameworks in their nature do things behind your back. Try to narrow this behavior as much as you can. Make things explicit as much as possible. Less to explain and document and more for your framework future compatibility.

Write? Rewrite? Use? - Thoughts on shaky frameworks

2009-09-22T14:11:00.000-07:00

Java is a great language. A great framework, tool, environment, libraries, all of that. From the very beginning the designers of Java thought of almost anything. Well, almost. ResourceBundle didn't support UTF-8 till Java 6! A few bugs were opened on that during the years (4093632 from 1997, 4221013 nicely written, 4749531 begging, 4919638 and 6204853 giving a final attempt, probably missing some). But it took a long time for someone to listen.

Were they drunk those guys? localization with support only for ISO-8859-1? are you nuts?

Yes, you could use escaped unicode (like '\uXXXX'), but it's not readable. Yes, you could use a UTF-8 file and escape your unicode as part of your build process (native2ascii). Some IDEs (netbeans) volunteered to do this automatically for you. But this is still bad.

Luckily for us, in Java 6 you can construct a PropertyResourceBundle with a Reader, thus you can create a Reader based on UTF-8. Or maybe not so luckily? The ResourceBundle.getBundle still works the old way, you cannot ask it to use UTF-8, so they hacked only half way - what the point? (Yes, you can use the Java 6 ResourceBundle.Control class to control the creation of your bundles but it's so lame. I implemented it and it works, but amount of code needed for such a simple task is unreasonable).

Here comes the question: should we use this problematic framework or write our own implementation? After all, it's not rocket science.

Usually I tend to be very careful with suggesting "write your own version of it". Our development instinct is usually seeing the flaws in some existing implementation, convincing our boss that we must write our own new version of it. In many cases our new version misses some important parts that the preexisting implementation already handled. In some cases the drawbacks that we saw in the preexisting code are in fact design decisions required in order to deal with scenarios that we don't think of or ignore and may tackle us later. So when I do see some existing commonly used library I'm very careful before I decide it's not good for me.

I found this suggested implementation. Seem much easier than the way to do that with the bone thrown at us with Java 6.

Still, might be not so bad idea to implement the entire thing, load of resource bundle, by myself. I still hesitate.

Technology Optimism

2009-08-14T13:38:00.000-07:00

At the time of massive cutoffs, transition of positions to low-cost centers and the sour smell of pessimism in the air -- I feel the spirit of Technology Optimism.

During the good times, the finest people with the brilliant ideas are sucked into the big corporations, tempted by nice salaries and all other perks. In many cases what they do in such positions is to work hard on projects that are later being shutdown. Let's admit, the big corporations are good at raising the money and selling the product, but when it comes to innovation, best innovative trick is to find the right startup to acquire. And this goes also for great innovative companies like google (GoogleMaps, GoogleVoice and many more are based on external acquisitions, see Wikipedia - List of acquisitions by Google).

There is a great advantage in a crisis, being a tool in reforming the market back to an efficient structure. We need more innovation and much more efficient companies.

Many brilliant people who have been back into the market lately are fed up working for giant dinosaurs, and start working on their own new startups. It may be hard at this point to raise the initial capital, but these startups are thin and efficient. And in a year or two we will start seeing the results with new technology domains that we couldn't imagine before.

When a bug makes you crazy

2009-08-13T14:15:00.000-07:00

This week I was sitting on one of those bugs which almost make you scream. Something is wrong, but you cannot find what creates this f-word behavior.

I was working on some web page example for some Taglib framework, but the page insisted on having an extra element that I didn't create. And when picked using FireFox, it pointed at a node in the XHTML that I'm definitely not rendering...

Step 1
Nothing in the code should make this happen. First suspicion is that I'm running an old instance, so I'm running a clean+compile and restarting the server, then when this doesn't change anything I'm adding a printout just to make sure. The new printout appears. Damn! I'm running the code that I'm looking at, but nothing there seems to be responsible for this result.

Step 2
I know that in order to debug I need a theory. But I cannot raise any theory, as clearly the result that I see is not something that is in the code... So I start playing (though I know it's not the methodic way for debugging, but I'm trying to play methodically). I find all places where I render this kind of element that is being rudely added where not belonged and I mark them each with a unique id. The id appears. Now I can identify the exact place that is responsible for this behavior, but I can't understand why it happens. An XHTML tag is added in a place in the document that surely I'm not printing.

Step 3
It took me some time. Maybe too much time. But I understand now that FireFox tree view of the page is cheating me... The element that FF is presenting at a certain place on the tree cannot be there. I'm not printing it there. Suddenly a strike hits me: GET BACK TO THE BASICS. Instead of using FireFox to locate for me the wicked node, let's view the source of the page. How simple and basic, though yet I spent about an hour before thinking about this simple thing.
And indeed, when viewing the page source I see that the problem is that I'm opening some HTML tag somewhere and not closing it, but FF closes it for me way beyond this point, creating a very strange node that I didn't create. In the treeview of the page FF presented a tag with a start and an end, while I accidentatlly opened a tag at some point and didn't close it. Now it's much easier to see the problem. And I also note that indeed FF showed that there are errors on page, with this error listed.

Moral

When the tool you debug with shows something that you cannot explain, go back to the basics: tcpdump, sniffer, printouts. The debug tool may be misleading you!
Believe only to bits and bytes, not to any fancy tool.
If you are working on a Web page, first look at the "errors on page", if you always insist on first fixing these you will save valuable time.
When a bug makes you crazy look him in his eyes, tell him that you will beat him. Don't show any weakness signs as these bastards know well how to recognize such signs.

IE is a pain in the ass

2009-07-02T04:35:00.000-07:00

Back in the old days when there were Internet Explorer (IE) and Netscape, everybody knew that it is very difficult to write something that would run perfectly on both. But since it was only two dominent browsers not many pointed the finger at IE. It was just that the two are different, life is challenging what could you say.

Today, when you have Opera and Chrome and Safari and FireFox -- ALL BEHAVING THE SAME -- and IE in his own realm, it is much easier to point the finger.

And here are some who do point the finger:

http://whyiesucks.blogspot.com/

http://iesucks.net/

http://zipthefuckup.com/2009/03/31/ie-is-a-pain-in-my-ass/

http://empireenterprises.com/noie.html

http://www.techsupportforum.com/design-forum/web-design-programming/111095-css-firefox-good-ie-bad.html

http://channel9.msdn.com/forums/Coffeehouse/427835-Why-is-IE-so-bad/

and there are more, just google for IE sucks or IE pain in the ass. Which it is, for most web developers.

It's nice to see that the other browsers do have a strict implementation according to the W3C standards, resulting with uniform behavior and ease of development. It says good things on the W3C standards being well defined and about the discipline of the coders of the other browsers, as opposed to IE.

Microsoft needs to drop IE altogether, start with a new code base and a new brand name, the current one is ruined.

JavaDay, JavaOne - Java Thoughts

2009-06-23T23:28:00.000-07:00

I was attending Sun's Java Day this week, here in Israel. The main themes Sun is pushing are JavaFX and MySQL.

Some thoughts on both, as well as other topics.

JavaFX

Sun's interest here is clear. Every handset or any sort of device running JavaFX means more licensing fees for Sun. In fact, JavaME is one of the only Java environments which Sun's really see money from, at least for now. This would probably change in the future with much more agressive sell of JavaSE licenses for extra features and support, but still handsets is a very important addressable market, competing with others technologies such as the simple browser, Android, iPhone Objective-C, etc. Sun interest is to have as many applications as possible running on top of JavaME, JavaFX and LWUIT (the last two are a bit competing on the same square). More applications means more consumer power (consumers who want to run these applications) thus more pressure on the handsets manufectures to have JavaME and JavaFX on the handset, droping the required license fee at Sun's pocket.

All fine till this point. But now it comes to JavaDay event, or JavaOne conference in SF. These events shall reflect what interests the Java community, be judgmental, compare competing technologies. But instead it becomes more and more a marketing event for what Sun is trying to push. Thus, JavaFX catch very high percents of all sessions (even a session on Maven had to be called "Deploying JavaFX apps with Maven" or something alike to get in...) And it is totally lacking the critic perspective, comparing with other competing technologies such as Android, GWT, zool, YUI, Adobe AIR etc.

Is JavaFX interesting? probably yes, but not at the amounts poured at us.

As for the judgmental perspective, I'm missing in JavaFX the separation between content and style, without that this would be a step back. Does JavaFX work with CSS? not that I know of. Is there a similar alternative? The declarative approach of JavaFX is good, but why the choice of JSON-like declarations and not XHTML? The language is very close to Groovy, but not exactly the same (Groovy is from Spring creators), is there a path to consolidate these two new scripting languages? And why should we go with a propraietary environment and language, and not with W3C standards adopted by the browsers running on mostly all handsets? ... Yes, I know there are answers for some of the questions above, but in the huge time dedicated for JavaFX, alternatives, as well as known flaws and drawbacks, were not discussed.

Which brings me to the thought that independent Java events, such as Devoxx, are a better meeting place for the community.

MySQL

This was another theme at the event. MySQL is a powerful tool, the regulator shall take it out of the hands of Oracle to preserve competition.

Java Reference App

Together with Ronen Perez, I was presenting a session called "Java Reference Application - putting all the buzzwords together" focusing on our selected libraries and tools.

Java Community and New Technologies

It's good to see all these smart people together. Being smart, people follow the trends and new technologies carefully, asking "what is it for me? how does it make my life easier?" In the past you could see people enthusiastic about any new thing, today you see people much more careful and reluctant. It's either that we are fed up with disappointments from previous "innovations" or that the pace of things require us all to be more cautious and conservative. This, together with the competition between technologies, requires new technologies to be well cooked and invest much more effort in getting us in.

Technological Singularity

2009-05-28T02:56:00.000-07:00

A singular point is a point in which the rules that define the spectrum around are no longer valid. In Mathematics a singular point is the point for which a function is undefined, in Physics, Gravitational Singularity means infinite gravitational field, known also to be a Black Hole.

Technological Singularity is the point of time when computers would be more inteligent than humans, thus creation of new computers and software programs can better be made by computers. Though sounds science fiction, something like the Matrix, I do know, as of today, of humans that are less inteligent than computers, one shall admit computers did part of the way.

The steps towards singularity in software engineering would be the creation of new software languages which are much more declarative rather than procedural. Computers would be able to get well-formed requirements, or constraints, to create the required piece of software. Prof. David Harel started talking about it back in 2000. Now the problem with our usual requirements is that they are not so well-formed, filled with holes and hacked with contradicting constraints. But what if a computer can identify the holes and contradictions and ask us what we really meant, then go and create the software. And maybe there is no need to ask us, since the computer knows better anyhow. The famous inventor and futurists Ray Kurzweil is speaking about technological singularity in his book The Singularity is Near with forecasts such as that by 2020 personal computers will have the same processing power as human brains and by 2030s Mind Uploading would become possible.

It's closer than you think. Declarative programming. Computers who understand requirements. Technological singularity. At which point the only occupation still relevant for humans would be lawyers, as computers cannot be mean enough for this job.

The future is surely an adventure. Some would see the upside potential of it, others may see the risks, which are there of course. For those and for those it would be good to read some advice from Daniel H. Wilson, on How To Survive a Robot Uprising: Tips on Defending Yourself Against the Coming Rebellion.

Maven, Static Code Analysis, Code Coverage and more

2009-05-25T17:01:00.000-07:00

Static code analysis and Code Coverage are becoming as basic as IDE syntax highlighting. Tools like FindBugs, Cobertura, Checkstyle, PMD, Eclemma and others, with plugins to Eclipse and Maven extensions, makes your life as a programmer more manageable, having Maven build a site for you with your code coverage results and following Checkstyle remarks in the IDE.

The title connects between things that are uncorrelated, except for being a measurement for modern Java development practicing. And of course Maven relates it all together also.

Today a Java programmer that knows only Java, even if very well, but intentionally ignores what's arround in the industry, e.g. sticking with good old Ant ("Why do I need this Maven, I'm doing just fine with Ant, just leave me in your mother") becomes a bit obsolete. The strength of Maven is that you can take a project with its POM.xml, and open it on another machine, even with another IDE that supports Maven.

So, to summarize.
As a programmer you are usually stressed and under pressure to close your tasks. Find some time to enhance your environment with the relevant Java Power Tools.

Wicked Problems

2009-04-12T08:49:00.000-07:00

Wicked problems in software design are problems that tempt you into a certain solution or design that seems reasonable and easy but turn out to be holey as a leach.

Mature developers and software architects know how to avoid wicked problems. Managers with lack of technical background may fall into such traps without even noticing and without realizing the trap even in retrospect, you may find a manager who would be very proud of a solution his group created for a wicked problem, investing in it a few man months, without seeing other alternatives of avoiding it or implementing it a bit differently with much less effort and side effects.

Straight solutions for wicked problems tend to be not only costly for implementing, but also for the long run, as the stitches sewed to close all holes tend to get unstitched, torn open.

Let me take you through a very small problem that I was dealing with, and turned out to be wicked.

Some background. In web applications we need occasionally to get info from the server according to user selections without turning to a new page. This is called an AJAX operation. There are several frameworks for supporting easy implementation of AJAX calls. One of which is YUI (Yahoo User Interface), which does on top of this all sort of other great things, supporting similar web experience on different web browsers.

YUI has a component for AJAX auto completion. The component is quite powerful and allows selections ranging from simple ones like this:

to more complicated, as this one:

In YUI the selection made is entered into the input-box and this is the information sent back to the server. We wanted to add the ability to send back to the server an ID instead of the selected value itself. This is useful since in many cases we want to present to the user meaningful text while the server really needs an ID. And we wanted it to work also for the multi-select version.

The solution seemed to be simply to add a hidden field and put the IDs there. The server would of course need to send in the AJAX reply the text to present in the list and input-box, as well as the ID per each entry. Making it part of the JSON reply is simple. Now, every time the user makes a selection we can add the ID into the hidden field, even for multi-select.

Simple and easy. We even found a few implementations like this on the net.

But this is not enough...

Since the user types his selection, he may type a selection without selecting (clicking on it) from the suggestion list. The user may also delete things after selecting them and then the hidden field needs to be updated. He may also retype characters into a selection made already, thus changing it to another, legal or not.

One thing came to our mind is that this feature needs to go with "Forced Selection" which means that the user cannot add things into the input-box without selecting them from the suggestion list. This may solve some of the problems. But not the deletion problem, we still need to take care of user deleting a value from the input-box. The only way to tackle the deletion problem is by populating the IDs hidden field only when the user leaves this input-box (when the focus is lost, i.e. in the "onBlur" event). Then what we can do is to loop through all values in the input-box, add a relevant ID for each in the hidden field, and display a warning for illegal values (for which there is no matching ID). The problem now is that at this point we may not have all IDs, so we need to go back to the server, in an AJAX call and ask for the IDs. And the values in the input-box must be unique, that is: each must represent exactly one ID (there cannot be two John Doe).

At this point we realized that we are going to work hard for a solution that would not bring much value to application writers, as they would need anyhow to implement a function on the server side for replacing textual values for IDs, and those textual values must be unique. So why don't just post the textual values back to the server as is, exactly as implemented originally by YUI.

Well, there is one reason for the hard work. And it is supplying warnings to the user while he types and not following post back. By checking whether all textual values has legal IDs we can warn the user for illegal values before posting the form. But then, one needs to decide whether this worth the investment, and whether it can work well.

We came to the conclusion that the people at YUI knew what they were doing. It is way too tricky to post back IDs. No, it's even more than that. It's wicked.

Now, we invested about half a day in investigating the problem and thinking of solutions, and then eventually decided to avoid this implementation. How many of us would do the same after investing a week in investigating?

A wicked problem is usually a problem that you'd better waive, avoid, abandon. Suggest other ways of doing it. Replace with another feature which is easiest to implement. The efforts of solving the wicked problem do not justify its value. Yes, a wicked problem is sexy, if you solve it you can become a true hero, it makes out a really neat feature. It is courage and experience, needed to realize the real effort for these wicked problems and to admit they do not worth it, even if some time was already invested in analysis and investigation. Or, to find the easier design, almost fulfilling the wicked problem, achieving most of the goals with much less effort.

Sun's Future

2009-03-29T04:03:00.000-07:00

Rumors on IBM about to acquire Sun has taken Sun's stock up recently. The idea of Sun being acquired were running around already for some months when Sun's stock was dropping to 10$ (e.g. here, from August 2008). Today Sun is traded back at about 8$, after jumping more than 60% in the past two weeks.

When Jonathan Schwartz, Sun's CEO, announced the change of Sun's ticker from SUNW to JAVA, at August 23rd 2007, that was the beginning of the end. The price of the stock was around 19$ then, after visiting 253$ (!!!) seven years earlier. The main problem is that unlike Microsoft or IBM, Sun never succeeded in making money out of software. They had a history of starting software products and then shutting them down, and they never really cashed the success of Java. Sun is a company with technical excellence and innovative spirit, but for the benefit of itself and of the software market, it shall find its place within a company that knows how to cash out these qualities.

-------------
May 2009:
Well, by now we all know that it was Oracle eventually to buy Sun. Still keeping the above relevant. I'm surly wishing all the best for this couple. And for IBM as well of course.
-------------

Software Factories

2009-03-16T16:41:00.000-07:00

Before the industrial revolution most of the products were made by hand, crafted. Is software manufacturing today more like a confection (ready made clothing) or more like "haute couture"?

The term "Software Factory" was created for describing the possibility of getting software requirements and sending them into a "factory" for manufecturing. (Do not mix that with a Microsoft technology with a similar name).

The challenge is that with software, customer needs may be very complicated. Can we just send the requirements into a "Software Factory" and get it done? (This in some way contradicts the idea of constant communication embedded in Agile Development.)

Think of airport control system that needs to address all kind of airports and equipment. Think of banking software that shall address different banks and communicate with all sorts of external interfaces. Can we create modules that are generic enough so that customization for specific customer needs can be done by cheap work force and without too much complications and investment?

Most software projects cannot be cloned. There is no customer that resembles another, requirements are different and the "physical" legacy environment is not the same.

Closets are also not all the same. Each customer may have different specific requirements. Nevertheless, most carpenters manage to stay scalable and address specific requirements by setting standards, such as several available sizes (you can order a closet of 2.0 or 2.4 meter height, however, 3.0 meter height would be a special order, would be processed uniquely and priced accordingly) and several available woods.

To achieve the ability of creating software as in "Software Factory", we shall:

Narrow the options and alternatives. State clearly what can be built as a standard and what is a special order.
Externalize a clear API
Insist on always getting clear requirements before starting (did you see a carpenter starting to work on a closet before having all measurements?)
Have core modules, easy to customize for very specific needs, with hooks and plug-ins for adding specific functionality
Create a very strict and close way of customizing the product, allowing it to be done by cheap work force. Always prefer customization techniques which require less thinking. Customization shall not be art, it shall be very technical and strict, preferably with clear instructions.
Both the core module and the customizable product shall be easily testable, each on its own.

I had a debate recently on whether the preferred technique for rendering raw data into HTML is JSP with Taglibs or XSL. Though XSL is very powerful and can do anything with raw XML at hand, I argue that a set of strict Taglibs with the instruction not to use Java code inside JSP makes it much easier to make this task in the mode of "Software Factory". In oppose to XSL, which is in some way an "art", using a strict set of predefined Taglibs is something that is much more narrowed, cannot handle all cases and special requirements, but can certainly provide most needs and most importantly, can be done by almost anyone, given clear requirements.

We are already in a period in which most organizations cannot afford having their software be "haute couture", they must settle on confection, ready-made, with some adaptations. Most software companies who would still manufecture their organizational software products as "haute couture" would not survive, unless in very specific and unique domains. When software becomes a commodity, like a cheap chip, the only way to produce it is in a "Software Factory" model.

Real Life Architecture

2009-03-13T00:49:00.000-07:00

When designing a system one shall think of all possible scenarios and error cases. Ignoring a possible scenario may lead later to all sort of problems, ranging from crashes to logical errors (e.g. fulfilling a financial transaction without finishing the order process, so the customer is debited but would not get any product).

One thing that we shall bear in mind is that listing all possible scenarios does not mean implementing a solution for all possible scenarios. There are error cases that are quite rare or not-so-rare but very difficult to handle, maybe even require a whole different design approach. A possible solution may be to log the problem to a specific database for manual handling, or for a later amending batch operation. Such approach can save a lot of the hassle during all stages: design, implementation and maintenance.

I may rephrase the above rule: listing all possible scenarios is a MUST, however, implementing a solution for all possible scenarios as an integral part of your system is probably not always the best solution.

Usually implementing a fully-blown solution for rare cases doesn't really work, as it is not tested enough for all variants and nuances of the exact scenario, and when the rare cases do happen you see that some modifications shall apply. Taking the treatment of rare special cases outside of the system, to either manual or batch operation may thus be a wise thing to do.

I call this "Real Life Architecture" based on the famous article by Gregor Hohpe Starbucks Does Not Use Two-Phase-Commit, which appears also as a transcript at Joel Spolsky's The Best Software Writing I. Gregor argues that in real life we often use asynchronous messaging without worrying too much about error scenarios, which we do deal with when happen, in somewhat batch way. As an example Gregor goes through the ordering mechanism at Starbucks, its simplicity and efficiency.

Be proud of your code!

2009-02-28T22:31:00.000-08:00

I'm usually quite proud of my code. If I'm not, I keep polishing it. When I review code of my guys, one of the things I insist of is to have the person's name on his code. Like on Dexter Horton Building.

If the guy says: "but all I did here is just to add this one little function".
"Well," I say to him, "so add your name above the function".

If the guy says: "but all I did was to fix this line. I didn't really added any new line of code".
"Well," I say to him, "so add your name above the line that you fixed, with a date and explanation of the fix".

But if the guy says: "hmmm, I'm not sure I'd like to put my name here. I'm not sure it works so well". Then I jump and say "haha! this is exactly why I insist on having your name there. Not initials. Not your nick name. A name that we can trace back when you will be a big manager at IBM, so you better don't leave any bug there."

The building above is real. I took this picture myself (I won't let you know where it's located. If you're curious you would probably google it.) I use this picture when I talk about code review and my insistence on having the developer's name on his or her code. Alex from mistralzonline, already documented this picture as part of his talk on present and represent, at his blog "On Everything And Nothing In Particular".

The risks of redundant code (or - Less is More)

2009-01-29T02:51:00.000-08:00

One of the preferred refactoring methods of my choice is deleting redundant code.

I separate between excessive code and redundant code:

1. Redundant code is code that never runs:

Unreachable code
Unused classes, interfaces and methods
Commented-out code
Code that looks wrong, then you understand that it is indeed wrong, but is being lucky enough not to be called
Code that deals with cases that cannot happen
Code that deals with cases that could happen in the past, but now cannot
Code that is there for future cases, which are not occuring yet (and may be added sometime)

2. Excessive code is code that runs, but can be reorganized into less lines of code:

Unnecessary abstractions, unneeded levels of delegations
Copy-Paste that can be moved to methods, inheritance, templates

Both are bad and should be treated. But the first category, redundant code, is much riskier. The main risk of redundant code is that it looks innocent and useful, but it doesn't really work and it is never tested. In many cases you assume that there is some value in this code, so you keep it around, maybe it will become useful sometime. Well, most probably it will become useful by creating new bugs when you accidentally use it. Remember: this code was never really tested, just being there, so familiar, part of your code, does not make it something that works. And in many cases it confuses you, as you believe that a feature is already written, because the code is there, you only need to pass another value or take some code out of a comment. But this code was not maintained or tested for ages, if at all! How can you rely on it? Why keep it there in the first place?

My recommendation usually is to be very cautious with code that never ran and is just lying there, as it may be lying and not doing what it supposed to do.

Keeping unnecessary code requires you also to maintain these lines of code that you don't really need. Again, I say - get rid of it.

Generic Software Mission Statement

2009-01-08T04:42:00.000-08:00

Following my previous post on the subject, I created the following generic mission statement:

Produce quality code, on time, for the success of the company.

Yes, I know, it's VERY GENERIC, but as a general mission statement for software groups it's quite good.

The problem with a more specific mission statement is that it may miss the responsibility of some of your employees. I had a few years ago a small group dealing with projects for customer applications. In one of our business group talks, the business group manager threw at the entire group that the goal for the team in the following quarter was to achieve sales target of 1M$.

The problem was that it was a group of programmers, not sales people, and achieving sales target was not part of our responsibility. The manager was just pumping down to us his own target (probably adding 25%, just to increase our motivation). The results of this talk was horrible, people were going around thinking how to bring 1M$, instead of doing their programming tasks. It scared some and it totally ruined the motivation of most, who usually do not relate their code to money. Code is art, relating it to money makes it dirty. Instead of speaking of money it was much better to speak about features, projects and deadlines. Even setting for a programming team a target of number of customer projects to achieve is wrong. Mission and targets shall be focused on the responsibility area of the team. It's the manager's responsibility to hide all the rest.

Vision, Mission, Goals, Objectives and Mesurements

2009-01-05T13:26:00.000-08:00

Year start is a good oppotunity to think about vision, mission and goals. I know that there are those cynical of you, who treat vision and mission as a waste of time. I tend to agree. It's a waste of time if you invest in it too much, and it may become a farce when you invest 15 minutes or more in explaining your vision or mission statement to your organization. Vision and mission are a great tool for saying something short and not too meaningful, but there are still important as they do shed light on your VALUES -- what do you value and where you are going to put your efforts.

The much more important are the goals and objectives. This is your opportunity to list the strategic achievements you want to bring. Goals and objectives should be SMART. That is: Specific (not vauge or too general), Measurable (state the measurement as part of the goal), Attainable (be realistic), Relevant (related to your vision, even if you don't have one yet), and -- Time-bound (otherwise it's like an exam without a time limit). There are, by the way, other interpretations for the SMART abbreviation but the main idea stays the same -- be clear and make the goals something that can be checked at a certain point of time and marked as done/not done.

SMART is smart to say but difficult to do, especially when you plan the future, which is what you do with setting goals. Let's take some examples:

For "Innovation", you may set the following goal: come up along the year with at least 5 new major (define major?) feature ideas, for which at least 3 would be judged as feasible and relevant to the market and at least one would be developed.
"Quality" is tricky. You may want to give bonus to the programmer who creates the fewest bugs, you may want to read before what Joel thinks on that.
"Investigating" new domain: what is the target goal to set? I usually set the goal as giving a lecture on the subject, at a certain agreed date. Usually it promises decent investigation of the subject. In some cases I add the requirement for documenting the material in our organizational wiki.

Setting strategic goals for the coming year/quarter/month is one of the most important tasks of a manager. Ask yourself, how you want your accomplishment powerpoint presentation to look like at the end of the period. Or maybe even better, prepare this powerpoint presentation ahead. Then set the goals and tasks that derived from it, aiming at minimum changes for your presentation.

Once you have the goals and objectives you can go back to the vision and mission, it would be much easier now to state them in 5 minutes. And if you don't find something strong enough go for "Bring qualitative value to the organization".

Test Automation ROI

2008-12-25T19:59:00.000-08:00

Recently I was working on a ROI (Return-on-Investment) Model for Test Automation. It was quite interesting to note that there are not too many "ready-made-excels" out there for this purpose. The ones that exist come mainly from automation tool manufectures. So I went to creating my own model, resulting with interesting notions.

Test Automation ROI Ingredients

Expenditure side

NRE (Non-Recurring Effort / Expense) - buying tool license, training people, raising the environemnt and infrastructure
Implementing the initial testing scenarios
Maintaining the testing scenarios and the environemnt and infrastructure
Adding new scenarios and new required features to the environemnt and infrastructure
Usually requires better trained testing engineers to maintain the automated testing system

Revenue side

In the long run, may be able to reduce manpower (less manual testing is needed)
Reduce number of escaped defects
Increase development confidence, allowing more features per release, thus less releases per year
Shorter test and development cycles and improved TTM (Time-to-Market)
Enhances the abilities of the testing team ("soft" revenue)

Influencing Factors

When coming to real life numbers it appears that it is very tricky to achieve positive ROI. And even if there is positive ROI the break even point comes after at least one year of investment, and in most cases two years. It's a simple case of big lump sum investment for future expected revenues.

The ROI is strongly related to the amount of bugs detected in the system in general and specifically in the field, number of existing manual testing engineers and the number of releases per year:

A system with very low amount of bugs that are mostly found at testing room and not in the field, would make it harder for automation to have positive ROI, as the potential of reducing escaped defects is low
If the current existing number of manual testing engineers is small, potential of revenues in reducing manpower is low. However, if number of escaped defects is high automation can assist in lowering it
Low number of releases per year would also make it harder to achieve positive ROI, as the lump sum investment is made for less cycles of use per year

The ROI is strongly related to the changes in the system:

Need for new scenarios and features in the environemt along the year means more investment in maintaing the testing system up-to-date
The lifetime of an automated scenario before it becomes broken or irrelevant because of changes in the system is very important

On an "average" system with some major new features each year, about 4 major releases per year and about 20 major escaped defects per year, following can be found:

In order to properly support and maintain a test automation project, you would probably need the same amount of people as you had before with manual testing, all along. The new features in your product requires new testing scenarios as well as new features in the test automation environment
The major revenues stem from reducing number of major releases (adding more features per release) and from reduced number of defects in field
It takes about 18 months to return the investment

Projects that are not stabilized yet and have too many changes would probably not earn from end-to-end test automation, although automated regression based on UnitTesting can still assist.

Module Separation

2008-12-17T07:07:00.000-08:00

Module separation is one of the oldest tricks in software engineering. As it is too complicated to attack a big problem, we split it and attack it in pieces. The pieces should eventually work together. We don't want to have two teams digging a tunnel from two sides of the mountain, missing the meeting point. In the tunnel case you get two tunnels which may have some value, with software you will just have a dysfunctional messy system.

The engineering task, some would say the art, of defining the modules and the interactions between them, is crucial for the success of the system. Do we need our modules to be stateful? Do we want synchronous or asynchronous communication? What would be the messages language between the modules, would it be programming API or a communication layer? On top of which protocol the messages would travel? And on top of all, which modules should we define? and for each module, which modules it shall be aware of and which it shall be totally agnostic and unaware of?

Usually we would like the communication between modules, i.e. the public interfaces that a module should expose, to be as little as possible. Operations or sub-operations that require much communication would be preserved in the same module while operations or sub-operations that require lesser amount of communication can be split. We should also like to have some hierarchy between modules so that not every module would have to know all the others, preventing cyclic dependencies is usually an architectural requirement.

So, what is the definition of a module?

Trying to create a definition of a module, or even better, a subscription, is not easy. Recently I was attending a session by Rick Kazman the co-author of "Software Architecture in Practice". Rick referred to this question, defining a module as "the set of tasks belonging to a unite topic, managed and built by an independent team of people." It is interesting to note that Rick's definition focus on the organizational aspects more than on architectural aspects of "what is a module". Rick argues that this goes well with Conway's Law stated by Melvin Conway in 1968, arguing that the required communication between two teams has a direct relation to the required communication between the modules they develop.

Create your own module

A module should have a clear definition of roles and responsibilities, both technical (what the module does) and organizational (who is responsible of defects found at this module). It should have also a clear and concise public interface. A module would usually be deployed as one unit, but in many cases several modules would be deployed as a unit together. Different programming languages would give modules different semantics, syntax and deployment characteristics, yet still some programming languages may not have the notion of modules in the language itself. And, not less important, a module shall be clearly defined as a set of tasks managed and built by an independent team of people.

How to do a 2000 pieces puzzle in a team

Doing a huge puzzle in a big team, does increasing the size of the team assist? Is there a number at which we will have a negative marginal contribution? ("Mythical Man Month"). Can you create "modules" here? This question about puzzle building is inspired by Chris Rupp's talk at JAOO 2008 (see video capture here).

It's a very nice exercise for a management course. I would think of the following design: (a) dispatcher team - goes through the pieces and assign them to the other teams; (b) teams by colors and shades, e.g. Dark Blue team would deal with the sea part while the Light Blue team would deal with the sky; (c) frame team, getting all the pieces that shall compose the frame. Each team is a module, so we have a dispatching module, a few color building modules, by colors and shades, and a frame building module.

In case of resource shortage we can settle without the dispatching team and let the building teams themselves do the dispatching. Number of people will define the separation into colors for the color building teams, but even if we would have 1,000 people we would not define 10 shades of blue, as the interaction between the teams (I got this blue piece but it seems unrelated to me) would be too exhausting.

Shall module definition be influenced by external factors?

We saw in the puzzle example that to some extent, number of available people would influence the number of shade teams that we may define. Expertise and skill set of our resources, as well as their geographical disposition shall also play a role here, but we will leave this for a later post on the relations between organizational structure and architecture.

Use Explaining Variables!

2008-11-30T01:15:00.000-08:00

Explaining variables are temporary variables that a method doesn't really need, but makes it more readable. I like them very much.

"Introduce Explaining Variable" is in fact a refactoring method in Martin Fowler's famous Refactoring catalogue -- p. 124 in the Refactoring book (it's a bit against Fowler's "Replace Temp with Query" -- but this is controversial anyhow -- I for myself like temp variables in any shape and form! As long as they have meaningful name of course...).

Why temp "Explaining Variables" are great?

It's better than a comment, the name of the variable says what you did in this funny math operation or complex logical act; comments are neglected sometimes, changing the logic and keeping the old, not accurate anymore, comment. The chances for a variable saying "isTheWorkerAvailableAndWithoutConstraints" to really mean what it says are much higher than a mere comment - we can count on the programmer that if something changes the name of the variable would change. Or we shoot him if not. With a comment we would feel bad to shoot a good programmer just for not updating a comment.
It goes anywhere. You don't need to try and recall "so where are we now" or "what did we put in this variable" - the long funny name says it all, and says it everywhere it goes!
The name can carry with it important info as measurment units. I'm extremely fond of variables with names like "timeoutInMillisec" or "distanceInMeters" or "radiationTimeInSeconds". It can save lives. Or satellites.
When calling a method it's sometimes more readable to send a temp variable with a decent name to hold the argument you want to send, rather than just send the argument. For example, a variable called "shouldSort" set to false may be much more readable than just sending "false". The programmer may also add, above the init to false, why false is the right value to send.

Usually during code review, when a small thing is not clear, I prefer an explaining variable over a comment.

Make sure to have a strict XSD!

2008-11-14T09:04:00.000-08:00

I'm going to tell you three things here:

Don't invent new languages!

XMLs are also languages (if they describe flow control)

If you do create a new XML based language, make sure that it has a strict schema!

-------

Do not invent new languages

We have enough of them. And by a "new language" I mean anything that describes a flow, that includes all kind of weired XMLs. In some cases people call these XMLs "configuration" but when I look into it, it's NOT. Configuration is when you want to set a value to some attribute. If it is a complex attribute, e.g. a network topology, you may need to use XML, as you need a way to indicate hierarchy. But when it comes to defining a flow, and you have conditions maybe even method calls and loops, this is a new language! don't underestimate it.

What's BAD with creating your own new cute language?
(I'd even phrase it: "Why DSL sucks!")

Suppose that you have a service and you want to allow the user to externally configure its flow. Now you don't want to expose your code and allow the user to change it (that would be indeed a weird idea). And you don't want to load in run-time some external lib or dll or java class, which the user can compile and add. Which again is a reasonable decision. You want to keep it as simple as editing a text file. So you create your own little XML based language and it works great. The problem starts when problems start. And it's not a tautology. When something is not working correctly with your cute little language you find it very hard to debug it, as you don't have a proper debugger, nor even a proper IDE for it, not to speak of reasonable intellisence auto-complete. Your cute language is orphaned, to properly support it you will find yourself investing huge efforts, much beyond what you have planned originally.

What's the alternative

The best way, if you wish to have external flow configuration that includes conditions and such, is to use some existing script language and call it from your program. Perl, Python, JPython, of course Groovy, and others, can be good. The advantage is that you get a mature language with all the required surroundings. You can run the script from within your program and can get back feedback on variable values or return code. And it is much easier to explain the language to your user, you just point him to the language tutorial.

But if you do need your own new XML based langauge

Make sure that the language has a very strict XSD:

Use enums for list of values, so the user cannot select anything, only what's legal

Use types: never allow to get a string for an int!

Use regexp to define the allowed patterns of string values

Don't use the same attribute for two purposes, it breaks the schema: if you can get either an int (number) or a string (variable), use two different attributes, each will have its own rule

Use different XML elements to oblige set of attributes: suppose that A-B and C-D are attribute couples (i.e. if I use attribute A I must also provide B, and if I use C I must provide D), it's better to break them into two different elements, even if they are similar in nature

A strict schema gives you a good language protection, thus less bugs, a decent intellisence auto-complete if you use a decent XML editor, and a clear documentation for the user of what's right and what's wrong.

The same as any respectable language has its BNF, if you invent a new XML based language you should base it on a strict schema.

* It's done already!

2008-11-12T21:20:00.000-08:00

Shlomi Hazan, a student of mine, was dragging his workshop for a second year. It was his last and final course. Finally, Thank God!, he submitted his work.

Here he is, with 2 laptops he brought with him, one desktop pc, one AudioCodes GW, a SIP phone and some POTS phones. He is in the computers lab, so though he did bring a lot of equipment, some of the equipment you see is part of the lab...

Now, I'm enthusiastic, not only because finally Shlomi decided to finish his degree and submit his workshop, but because it was a very nice workshop without too much code.

Shlomi presented a SIP Conference Call server. The thing is that using Asterisk you can get almost all of it done (and if you need for some reason a JAIN SIP stack, you will find an open source for this as well).

I think that asterisk is an amazing open source project.

NotifyingBlockingThreadPoolExecutor

2008-10-23T05:49:00.000-07:00

I've found this interesting article in java.net, extending Java's ThreadPool.

Don't Make Me Think!

2008-10-12T17:59:00.001-07:00

I'm reading the great "common sense approach to Web Usability" - Don't Make Me Think book, by Steve Krug.

And I recall that I have the exact same message when reviewing code: Don't Make Me Think. If I have to think more than a second on a line of code it means that it's too complicated, you need to break it to two, maybe add a good comment above.

If I start asking questions like "why did you do this or that?" it's not good, it means that your code is not self explanatory, or that you had a good reason for doing something in a special, maybe unorthodox way, or just not the way I'd think of doing it, but you didn't explain it well enough in a comment.

When reviewing code I usually ask the developer, "OK open me the IDE, now let me try to understand by my own", "don't explain anything". "Shut up, stop explaining what you did, don't you understand I want some quite?!"

Then if I understand all by my own without a need to use my brain too much it's a good sign. The programmer gets his candy and we say goodbye. Otherwise the big refactoring, commenting, arranging work starts. Together with some oriental songs mumbling.

Selecting the Right Technology or - the Art of "Satisficing"

2008-10-10T16:10:00.000-07:00

These modern times are unmerciful. Back in the old days, when you had a programming task you could select between punching the pink punch-cards or the yellow punch-cards. Later, when Java came-up, if you selected Java, the library selection was quite easy: you went with what Java is providing – this was in fact one of the greatest things with Java – its built in set of common libraries, so there is one standard for IO work or Database connectivity. Any new programmer that comes along either knows already the standard or he should learn it. But no smartass would come knowing some all-different thing, asking dumb questions like “why are you using this and not that?” – luckily for us, there was only one option for everything.

But we are now at new times. For every fart you want to make, there are several technologies you may use (pardon for the word “technology”). And selecting the right one is in some cases more troublesome than just doing the damn thing.

Selecting the right technology: well, here is the “full process”:

Divide the thing you are going to build to relevant “topics”, each may need a whole different another set of technologies – you are anyway going to divide your project to modules or tasks. “Topics” may include: logging, configuration, communication channel, XML parsing and manipulation, database connectivity, caching, pooling, redundancy and failover support, build infrastructure, continuous integration support, unit test framework and code coverage support, static code analysis tool.

Build the list of ALL relevant technologies, for each relevant topic – don’t miss any relevant technology as it might be the one for you!

Learn all technologies: write some code with each of them or at least read some example of code to see how much you like it; read analysis of pros and cons of each technology or do the analysis yourself; read some comparative benchmarks or better run them by yourself.

Create a nicely built comparison table for each topic, with all relevant comparison categories and aspects, create a team to discuss the comparison categories and give them weights, then give a (subjective) grade for each technology on each category and calculate the final score. If money is involved, add it to the equation somehow (not so simple, but with small amounts usually just added as another row to the comparison table with some grate from 0 to 10 or alike).

You have a winner, start working with it.

From time to time (once in a few months / once a year) conduct the process again. You cannot of course just rest on the laurels. New technologies keep coming, so you need to go again and reassess your conclusions. Of course, you may need to give some weight on “knowing already this technology” or “paying already for this license” or “doing the work already”, but it doesn’t mean that you would not choose to replace it after all with something better, maybe something with much better performance, nicer user interface or easier maintenance.

This is the full process. The problem with the full process is that it is VERY COSTLY. To conduct it thoroughly you may invest in the selection much more than the gain between optimal and random (or a better “smart random”).

The “full process” is probably the way for decisions with some several millions on the stake. And you would probably never go to #6 above, you would rest on your laurels after a decision is made, even if a new promising technology is just out – they always appear and it’s a never ending race, like waiting for the best electric equipment to be a bit cheaper, once it’s there you would wait for the better new model to hit the same price. This is the road for never deciding and never taking an action, or taking a decision but always changing your mind (a road that leads eventually to a dead end dwelling a lunatic house).

It’s simple economics, when you search for the cheapest Plasma TV-set, you MUST take into the equation the time spent and the fuel burnt. You may saved 100$ by investing 5 hours on the internet, 2 hours driving to the warehouse realizing that the thieves over there suddenly have a different price (“no, the price we gave you over the phone was for the TX model with no screen, you need the TWX which comes with a screen and is 400$ more, sorry!”). Eventually you get the TWX model that you want at what may seem as the real minimum price. But was it optimal?

“Satisficing“ is the keyword here. This word, invented by the economist and computer scientist, Herbert Simon, who is the only person who won both Nobel Prize and the Turing Award. It refers to the act of satisfying by selecting a suffice choice -- not necessarily the optimal one.

How to Satisfice?

Make sure that you do not miss the major relevant technologies, those that everybody talks about. For this you MUST make yourself well familiar with the subject.

There is a small chance that some un-spotted niche technology is your optimal choice, but since not many selected this niche technology, most chances it has some flaws. Missing a decent audience is a flaw as itself, which may affect later support, attention and availability of practiced manpower for this technology. So it's not a mistake to narrow the list to the major technologies that most people talk about and have the biggest communities, as long as you identify the domain correctly.

A big community using the technology usually means that this technology is flexible and versatile enough to accommodate all, which is an important sign. Your needs would probably also grow and differ along time, you cannot fully foresee it now. Using a flexible enough technology is the cure for unexpected future needs.

In many cases it's not a clear cut, nor is it a Black-n-White. You checked all aspects, consulted with colleagues and still hesitate -- flip a coin.

Standard protocols and APIs are for your rescue. If you have selected the wrong choice. A real crappy implementation. If it is based on a standard protocol or API you can just replace it with another implementation. Thus, do not tempt to proprietary extensions, stick to the standard. If there is an absolute need for using some proprietary extension (big efficiency boost or the only way to achieve something) narrow it to the minimum and isolate it in the code to a central place (e.g. one utility class which is the only one using the proprietary extensions). This way, if there is a need to change the implementation it would be easier to just re-implement the utility class.

Learning curve should be a factor. But you should remember it's a one time cost, compared to using the technology and maintaining the code, over several projects. Usually after you learn it, the complicated stuff becomes much less complicated not only for you but for the entire team, you find the way to wrap the complexities and make it easy to use. Give credit to the community, if it is said that something is good, wait before disqualifying it for being too complex.

Stick to things that work. If you see an example of two technologies working together (e.g. A and B) and a second example of another couple (C and D) but no other combination (no A and C or B and D), maybe there is a reason for that. Be caution before rushing to un-explored combinations.

If you do want to be on the edge and rely on un-explored territories (something without a clear and exact example) -- build your own simple example before taking the technology to the real battlefield of your full-blown project. And be nice, publish your example!

Consistency and uniformity are valuable! Try to select the same technologies for all your projects, thus you can share code, knowledge and bug fixes. Think twice before deviating to a new technology with investments in old substitute. Then think once more. Usually upgrading to a new version of a technology or moving to a whole different one is something to be done once in few years. If it happens to you twice a year check whether one of your programmers is checking all available technologies for some Academic assignment he is working on.

Be reluctant to adopt newly presented technologies in the market, even if everybody says it's the great new thing. Being on the cutting-edge may sometimes bleed! Technologies tend to mature, let others use the alpha and the beta. Wait for the 1.1 version. If you do want to take the new beta technology, make sure to test it well on a demo project which has the relevant features you need in your full-blown project.

Documented technology is always better than undocumented one. Prefer to ignore the less documented, even if it's cool.

Performance is important but is dynamic. Changes in HW and Computer Architecture (e.g. moving to multiple core CPU) may make a dominant choice descent. Updated versions of the implementations may also change the balance. Thus, usually if there is no real significant difference, of a multiple factor, performance should not play a distinctive role. Standard Protocols and APIs are more crucial.

Don't feel obliged to wait for a new version or a new technology that is about to come out, just because you want to explore it. It's like waiting on a new electric appliance model, there always be a new one coming shortly, for which you will feel again obliged to wait. Make your selections and go on.

In many cases it's not a clear cut, nor is it a Black-n-White.
You may feel here a deja-vu. Yes, I've said it already. But you must bear in mind that in many cases indeed, there is no right answer. This is another good reason to be conservative, keep with old technologies for which you have the knowledge and expertise, wait before adopting the cutting-edge, prefer documentation over nice fancy woos, prefer main stream over niche, prefer standards over proprietary.

Count on your gut feelings. But only after you do all the required investigations and exploring. Write some code, check the alternatives. But if after investing the effort you don't feel good with it - don't use it.

RM-788, Knowledge Sharing and Wikis

2008-09-25T06:18:00.000-07:00

I have a new "all in one" remote control. You get with it a small manual with instructions and operation codes. These things work that way: you need to set it to your appliances, so there is a specific code for each brand and model and you have to punch in the right code into the remote control, then if the code was accurate, you can start using the remote. In order to adjust it to the specific brand and model of your appliances you have to look for the appropriate codes, listed in the manual. So far so good. But it turned out that my TV set, PILOT of some unknown model, doesn't go with any of the codes listed in the manual for Pilot. One code did work, but only partially. Annoying.

What would you do at this point? One option was to return the remote to the store and make it ship back to Taiwan or China or wherever it's manufactured.

But then I remembered the great invention of the Internet. Maybe it will come to my rescue. So I Yahooed and Googled for the name of the remote control, RM-788 of Universal Electronics. And I got to some reseller sites, and even to the site of Universal Electronics themselves. But apart from seeing the picture of my remote there was no useful info available.

So I came back to the remote and to the manual. It appears that there is an auto-search function of the thing, which is a bit troublesome, being not really automatic, more like “semi-automatic” – the remote pass through all codes, but you have to check each code manually and then if not OK, you need to press the “Power” button to move to the next possible code. Since there are about 400 possible codes for TV sets I was a bit pessimistic. BUT, it did work! After a while I reached code 138 which worked perfectly for my Pilot TV (there is a way to know the currently set code, by counting number of led blinks for each digit, so the moment I got it working I wrote down immediately the magic number 138).

Now comes my social responsibility. I wanted to share this knowledge somewhere, for the benefit of other Pilot-TV owners who may try to use this remote (even if one in the entire universe!). Needless to say that the manufacture site doesn't allow to add comments or tips. Probably they are afraid of obscene language comments, and don't have the resources for an editor. (Do you know of any company that do provide such a “drop a tip in my site” service?)

There was one site on electronic appliances which did have the ability to add tips and even had the RM-788 remote listed in his catalogue! Unfortunately when trying to submit a tip the site ask me to register and fill in a ton of details (stuff like my mother maiden name, oh madden!) so I left this unwelcoming site in a hurry and rushed to my blog.

Which made me think -- We need a Wiki here.

Just to clarify the terms. Speaking about Wiki and the Wiki idea, I refer to the concept of free editing with immediate publication. For some nuances a few operations may require prior registration, but not for adding or editing content. When editing is allowed only for the site owner, or is not immediately reflected in the site, then it's not Wiki for me, it's just a Content Management System, based maybe on Wiki framework and syntax, but not on the Wiki idea.

The Wiki idea was invented as early as 1994 (see: http://en.wikipedia.org/wiki/WikiWikiWeb).
It came into wide notion with Wikipedia. Followed by many other Wiki sites, some as part of the WikiMedia foundation, others include Star Trek Wiki, World of Warcraft Wiki and many more (some lists can be found here, and here, and here, and here, and of course here and here).

Why Wiki is the best tool for knowledge sharing?

Because it allows someone with good will but no patience for bullshit to just get in and share his knowledge. Publishing the submission right away forces the submitter to pay more attention to his submission. Usually one would invest more time, when knowing that no other person is responsible of editing his post and fixing any spelling mistakes. Of course, someone may need to follow new or edited articles and validate them. But we don't have to wait for the validation in order to see how the new contribution looks, publicly. This is the Wiki idea!

The fear of vandalism, by the way, is almost irrelevant, when it comes to internal knowledge sharing inside organizations. This is the reason Wiki becomes more and more as the method of choice for knowledge sharing internally inside big corporations.

As for the RM-788 Pilot-TV code, it appears that there is a Wiki for Product information, called simply ProductWiki. It's commercial (not a non-profit). I would have drop my tip there, but it didn't have the spoken RM-788 listed in its catalogue, and required me to register for creating it, so I quit.

For more on Wiki as a tool for knowledge sharing, and other good references for managing information in the Wiki way:

http://www.ikiw.org/
-- and specifically: create-a-participatory-knowledgebase-on-a-wiki
http://www.wikipatterns.com

Google open sourcing their new browser, Chrome

2008-09-02T09:02:00.000-07:00

Google new browser, Chrome, is the talk of the day. And the comics are indeed great.

Some were enthusiastic about Google opening this vast project as an open source. Noble indeed. Google themselves are feeling virtuous about it, or at least want us to feel that way. After all, a great tool like this (no one really seen it yet, but they do know how to create a buzz and I guess it will be indeed a good browser) – by releasing this great, superb, new-era browser as an open source, anyone can take it and adapt it to his needs, even Microsoft can, and maybe will even do.

So why did they do it? Why to open source? You can give it for free without opening it...

Slide 37 lists the noble arguments.

But could they really?
Do they have the option the ship a proprietary browser?

NO, because they don’t have one.

This new Chrome browser is said to be built upon WebKit and Mozilla projects. Both projects are open sourced and require derived work to be open as well (section 3.2 in Mozilla Public License and 2 of the LGPL).

They could have find a way to keep some of the code closed, like maybe the V8 javascript VM. But it would probably be too risky legally. And they might need their lawyers ready for Android issues (or for buying Sun).

Bottom line:

No need to be too cocky about open sourcing. As nice as it is that you open it, you probably had no other choice, other open source initiatives worked hard on writing parts of your code.

CPU and GPU, from nvision 2008

2008-09-01T04:17:00.000-07:00

Nvidia had a geek party in San Jose, called nvision2008.
In one of the peak events of the conference, Adam Savage and Jamie Hyneman came to demonstrate the difference between a Central Processing Unit (CPU) and a Graphics Processing Unit (GPU), using color cannons.
(See: http://www.nvidia.com/content/nvision2008/day3.html).

Cory Ondrejka, former CTO at Linden Lab and one of Second Life founders posted the following videos from the party, showing the color cannons in action: collapsing geography: geek celebrity. Make you understand the power of GPUs. An inspiring color cannon! Wow!

Adding abilites to Java's ThreadPoolExecutor

2008-08-29T02:52:00.000-07:00

I had a discussion on Java's ThreadPoolExecuter, which seem not to have the ability to block producer from adding tasks when the thread pool's queue is full. I went through some possible solutions and I'm gonna write something about it, so please stay tuned (and if you have something to tip on this issue, say it now).

-----------------
Happy to update:
I've published it, in java.net, it is called Creating a NotifyingBlockingThreadPoolExecuter.

Google, on the way down

2008-08-28T11:28:00.000-07:00

I host my blog at blogspot, as you can see. Today I was editing the draft of the post "ChangeLightBulbWindowHandleEx" when suddenly, between one automatic save to another, the draft just disappeared. Vanished. Gone. I was left with the title alone, no content.

There is no versions saving in blogspot, so I couldn't go to the previous version. Then I thought maybe I can turn to blogspot support and ask them to recover my draft, they might have it there somewhere. Long shot, but worth trying.

I started to look for the e-mail of the blogspot support.
And, after a while I found the link saying Contacting Support. Great! So I followed it, to a nice page. And another nice page. And another. None of which offered an e-mail or a submission form. All tried to suggest advices for all sort of problems. There was a link leading to Blogger Help, but there again, no e-mail or decent bug report form. BLAH!

Then I started looking maybe someone already bumped into the same problem, maybe there is a hope somewhere. And I got to this mailing thread at the Google Blogger Help Group.

It could have been funny if it wasn't sad.

Don't you care about your bugs?

I know it doesn't really say something about the business model of google. But it says that they lost it somewhere down the road. Don't you have enough resources to care about your quality?

If I'd have google stock I would sell now.

ChangeLightBulbWindowHandleExfor Blind Catalan-speaking SpaniardA response to mistralzonline on "Why is that?"

2008-08-28T08:19:00.000-07:00

My friend Alex Romanov, in a recent post at his blog, asked why Nokia is not supporting a synchronization tool for his Mac, while he could easily find such a free tool on the web.

The answer for this question is found in the known article by Eric Lippert, “How many Microsoft employees does it take to change a lightbulb?”, making the math of how much resources are needed for a formal release of a simple thing. And since it takes a lot, you need not get off your course. Which can easily happen.

To summarize, the answer is: Focus.
Choose your lightbulbs!

I was once responsible of a State Machine module for a communication component. With the State Machine module we were providing also a nice editor tool, based on Microsoft Visio. The users liked the Visio editor which made life easier for them. We even thought of adding debug options to this editor to allow the users to run in debug mode in within our Visio tool. At some point I decided to stop the development and support of this tool and instructed all users to start writing their state machines in the raw XML that the module itself received as input. It happened when I realized that we are spending too much time on this nice utility. Every new feature in the State Machine module required a parallel development for the Visio tool. Too much support effort was invested in negotiating features with the users and bugs with the testing team, training and support, keeping up with the Visio versions and more. I realized that we do not have enough resources to support all of this (the resource was in fact one person, supporting by himself both the State Machine module and the Visio tool…, it was, by the way, the man behind mistralzonline from above, so he should know!). Focus is the word. When we realized that we cannot do it all at the required level of quality, we had to cut. And it’s better to cut the utility Visio tool rather than the module itself.

When Nokia decides not to provide the tool for synchronizing one of their models with Mac, it’s because they believe that with the resources they would put in it, they cannot decently do a good job. It’s not only development, it’s the entire life cycle, including support for users who get stuck (and do it better than google, a post on the horrible google support will follow soon).

Now, Nokia has the option of managing a community or at least pointing to the available solutions on the web. After all, there was such a free tool out there. It is interesting to note that not many commercial companies officially manage such a development community or point at available supporting tools related with their product. Sun does have the java.net, but it’s more like an open source arena.

Why is that?

Because for Nokia, pointing at a tool, even if accompanied with the common disclaimer waiving any liability or guarantee, still means taking responsibility. If the tool infringes privacy, causes loss of data or just simply doesn’t work right, the furious public will come to Nokia. To take such a responsibility Nokia would have to thoroughly test the tool, which brings us back to square one. This is why they choose to refrain and let you pick it up by yourself, which as appears you can do quite well.

Rolling the Dice - the Cube Project for Silverlight

2008-08-26T07:29:00.000-07:00

I am teaching a "Product Workshop" course at the MTA college in Tel-Aviv.
Usually students submit something that can be used as a "black box" product, like: a web application managing event invitations, a simulator for physics rules and many other nice things.
Whenever a student wants to create a library or a toolkit (that is, something to be used by other applications), I urge him or her (well, it's a him, let's stop with this political correctness, when it is a she the product is a very nice well designed, all screen pink, web application for match-making...^*), well, I urge him to put it out as an open source.

So here is something recently created by my student, Itamar Kotler:
http://www.codeproject.com/KB/silverlight/CubeProject.aspx

It's a component for creating a rotating cube out of 6 photos, right into your web site.

Funny that it is not inherently supported by Silverlight (there is something in Silverlight but the argument is that it doesn't really support full rotation through all possible angles). I guess it will have soon a decent substitute as part of Silverlight, which will probably disappoint Itamar. But then again he can always argue that he drove Microsoft to implement and present a decent rotating cube.

^* Just before I'd be accused of being a male chauvinist swine, some of the best projects I get are from women. And I can live with pink screens.

Do you have an API?

2008-08-01T06:18:00.000-07:00

This is so basic, yet still ignored so often.

When you buy a SW tool or thinking of developing one, one of the first questions to ask is: is there some kind of API exposed? CLI (Command Line Interface)? Web Services Interface?

A product without a clear good API means that you cannot customize it to your own needs, it's a "take it or leave it". Of course, you can ask for features and then wait in line.

A good API must allow to preseve states when needed. Suppose I want to perform a series of operations, like for example a chain of queries, I need to be able to send the 2nd query for filtering the result set without reminding the tool what was the first query. And the results should probably be cached for a specific amount of time. So I'd probably need some session ticket, request Id or alike.

With a good API I can use the tool in things that were not originally planned, and still do well. And when a question arises: does the tool know how to do this and that, the answer would much more easily be: yes, it does have the ability to programatically allow it on the client side.

Now if the product doesn't have an external good API, it probably says that it is not well designed with a clear separation of a Model View Controller (MVC) architecture.

Bottom line

Even the most graphical product MUST have a model and an external API.
Start always by thinking on the API or CLI version and ignore the presentation layer.
And think twice before taking a product that can be opearated only via its specific UI environment.

Think - MVC, MVC, MVC, MVC, MVC, MVC, MVC, MVC, MVC ...

To AJAX or NOT to AJAX?

2008-07-21T06:26:00.000-07:00

I had today a discussion on when to use AJAX and when not.

The idea is simple: the plain old request-response is usually a good thing, if you keep your pages small, which you should (CSS and JavaScripts should be external, thus a nicely built pure HTML page should be around 10-20K, not more).

The plain old request-response keeps your url meaningful - the state of the page is reflected by the url, which is a very good practice to follow.

The plain old request-response keeps your Back button function as it should, without any unnecessary tricks.

The plain old request-response keeps all your content available for search engines, which is usually what you want.

The plain old request-response keeps your user using what he is used to. No surprises.

So - when should AJAX be used?

Auto-completion
For fetching partial long lists based on some initial input
Quick response to something insignificant, like showing the current results of a poll right after your vote is done, blocking you from re-voting (not that you cannot delete your cookie or session identifier and go back to the page to vote again, usually the vote is not registered per IP address)

Some additional insights on this topic can be found here:
http://webdesign.about.com/od/ajax/a/aa092506.htm

Building features organically

2008-07-20T23:16:00.000-07:00

I ran into this interesting post and wanted to share it with you:
http://marcelo.sampa.com/marcelo-calbucci/brave-tech-world/Building-features-organically.htm

This is how it begins:

One of the most interesting aspects of Agile development is to not build things
that you won't use now or over-engineer a component because "we might need this
in the future". That is a wonderful thing for developers and it does cut a lot
of complexity and time to deliver the feature, by consequence making it less
buggy.

Now, the rest is quite the same, so you can either go there or not.
But the first comment (and only one when I visitied) is pretty smart. And counting on you lazy fellows that you might not invest in following the link and scrolling down, here's what Mike comments:

I spec from the "top down" and build from the "bottom up". That way, I think I
have a more throught-out overall plan, and it eliminates implementing some
possible dead-end features. Specs are much more malleable and have far fewer
requirements from a quality point of view. So I find it worthwhile to be a bit
more expansive and "future thinking" in a spec. But when it comes down to
implementation, I try to do as you say - and build from the bottom up with the
minimum set of functionality that can be made to work consistently. You can then
decide when you've "fiddled" enough, and decide to ship the features to your
customers.

Well said.

Thinking Procedural From Object Oriented and Event Driven Back to Procedural Programming

2008-07-03T23:43:00.000-07:00

Object Oriented Programming

Object Oriented Programming fits most of the applications we create today. The idea of methods or services, encapsulated into classes or components, with clear separation of concern, isolated from each other, is clear and obvious. This is the heart of the Object Oriented concept, of Component Based Development, of SOA, etc.

Event Driven Programming

Event Driven is another concept, raised mainly from User Interface applications, but also protocol stacks, state machines, message brokers and all kind of listeners in general, can be seen as based on event driven concepts. Event Driven does not necessarily relate to Object Oriented Architecture. But yet, it is not a simple procedural programming.

Other Non-Procedural

Other types of programming methods which are non-procedural in their nature may include Pattern Matching Programming (e.g. XSLT) and Declarative Programming (e.g. HTML, SQL; some view Object Oriented Programming as declarative, I disagree).

Thinking Procedural

All above - Object Oriented, Event Driven, Pattern Matching, Declarative Programming - keep us apart from the flow. We get used to ignore the flow, the entire end-to-end scenarios which eventually creates the system.

Long time since we created a pure procedural program. This made us stop thinking in a procedural way. Which is problematic, as eventually the program will run as a sequence and we have to make sure it will work properly.

Here come to rescue: Sequence Diagrams, Scenarios, XP User Stories, UI Storyboards and Test Driven Development. They were all invented to assist us solving the problem of forcing us back to "thinking procedural".

Bim Bam BOM

2008-06-30T04:19:00.000-07:00

We had this strange bug recently, while trying to parse a perfectly healthy XML file we got an exception saying:

org.xml.sax.SAXParseException: Document root element is missing.

We could, however, open the XML file in the browser without an error. And also our XML editor took it without a problem. Yet, in our code trying to operate some XSL transformation using Xalan, we got the exception above. And the problem was not with the transformation itself, at least according to the exception. The problem is with the XML document starting without a root. Though the first character seen in the file was an opening triangle bracket...

Well, there are things beyond what you see.

To find out what is the exact stream of bytes that the transformer receives, and doesn't like, we added the simplest debug line, dumping the bytes from the file to screen, not as chars but in their value. There appeared to be two bytes before the opening triangle bracket of the XML: FF FE.

At this point, my friend and colleague Effie Nadiv (famous for his Hebrew site, and a UI authority and legend), shouted out: it's the BOM! Xalan doesn't recognize the BOM correctly!!

And without further ado he presented the file (same file) in text mode and in binary mode:

See the FF FE at the beginning? This is the BOM.

BOM stands for Byte-Order-Mark, added to UTF-16 documents to denote the order of the bytes in each two consecutive bytes creating a character. UTF-8 documents may also have BOM, but it will be redundant and have no mean.

To read more about Unicode, UTF-8, UTF-16 and BOM, you may want to go to:
http://unicode.org/faq/utf_bom.html
http://en.wikipedia.org/wiki/Byte_Order_Mark

But, before you rush to the above, a GREAT reading material to understand once and for all the entire encoding and charset thing:
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky.
This is a must read!

Specifically to solve the above problem we changed the doc to UTF-8 without BOM. Most text editors support conversion to different unicode transformation formats, and allow the user to decide whether to add BOM to UTF-8 or not (probably it's better not to add, just turn off the option underneath).

=========================================
Added 21/7/08:
------------------------------
Just found this old newsgroup entry on the subject...
http://biglist.com/lists/xsl-list/archives/200208/msg01302.html
=========================================

Tag, Log, Debug!

2008-06-26T05:53:00.000-07:00

I was presenting with my friend Alex Romanov our work on "Automated Log Generation and Analysis using Collaborative Tagging" at the IBM Programming Languages and Environments seminar 2008, yesterday (25/6).

This is Alex with the poster describing our work:

More info on this work can be found here: http://ed.finnerty.googlepages.com/taglogdebug.
Here you can find the initial paper.
And finally, a blog that will follow our work was opened here: http://taglogdebug.blogspot.com

Eclipse Plugins Tutorial (OOPSLA '07)

2008-06-24T06:18:00.000-07:00

I was asked where can people get the materials from the tutorial on "Creating Plug-ins and Applications on Eclipse Platform" that Alex Romanov and I gave in Montreal on OOPSLA '07. It was while ago, and the materials are on the web already. But here are the links, as it seems google is not doing a good job referring people to the googlegroups site we created...

The tutorial deals with all the important things for getting started with Eclipse plug-ins: creating a plug-in project, the components of a plug-in project, GEF, MVC in the Eclipse Plug-in architecture, SWT, Actions, RCP and more.

So here is the companion booklet.
And here are the slides.
Both can be printed or redistributed, as is, for any purpose, referring back to the source.

Hope you can find it useful.

False Requirements

2008-06-11T23:40:00.000-07:00

Suppose you are assigned to design a system that should carry out people from place to place, and occasionally let's say once in ten years, would have to travel a giraffe. It's tricky, but you may be able to come out with some strange car with a very high ceiling, working out the balance and stabilization. It may not be so economic, but who is going to be piker when it comes to carrying giraffes. (One can of course suggest a car without a ceiling at all, which might be a good solution, but one of the other requirements rejects a convertible.)

It turns out that we are often bending and twisting simple systems just to carry the giraffe. And in many cases when digging into, we find out the giraffe was not even in the formal requirements to begin with! It grew in somewhere, in one's imagination, and became an important part of the system. Oh, how much we could have saved without this giraffe, and the system could have been much simpler...

Let the giraffe travel on its own!

If it was in the original requirements, go back to the system analyst or the guy who wrote the requirements and ask him: do you really need this giraffe thing? maybe we can send him with another vehicle?

Giraffes are nice, but don't let them into your system.
Unless you want to run a zoo.

What would you like to know about your candidate? (Or - how can you know that he is Smart and Gets things done?)

2008-05-30T06:32:00.000-07:00

After my session at JavaDay I had a conversation with a colleague from Sun on technical interviews. We started with the question whether a Java candidate should be in the details of Generics (e.g. know the term "erasure" and what it means). I'll touch this in a moment but let's begin in an ordered manner.

What do you need to achieve in an interview?

The famous answer comes from Joel Spolky's "Smart & Gets Things Done" known article (it has an updated version, but I like better the original one. And it got out by now also as a book carrying the same title!) Sorry for the massive PR for Joel (he is NOT my cousin), but this is really a great source to begin with.

Following Joel, let's just mark out that you need to achieve two things in an interview:

Attract the candidate: sell yourself, your company and the offered position. Make sure that if you want this person you will get him.
-- This is highly important and sometimes neglected (e.g. making the candidate wait for his interview for 30 minutes), but we will keep this for another post.

Make a Hire / No Hire decision (is the candidate Smart and Gets things done?)

__________________

What do you need to know about your candidate?

Communication skills - can he explain himself well. Ask him about his previous job or big project at college, see if he can well explain what he did.
Does he see the big picture? - continuing with his big project, make him explain what other did in the same project, ask about other alternatives that were there to implement the same thing and why the one taken was selected.
Technologies - ask about the technologies used in his big project and what can he tell about each of them.
Design - you can learn about the candidate's design abilities by talking about the design of his big project, but it would be better here to tackle him with something he haven't thought about or at least didn't talk about in his previous interviews. It can be a pretty simple design, like some kind of LRU Cache, or Copy on Write mechanism, or anything else you have in mind. But make sure to listen - ask shortly, give enough time to think and then listen. This is a good opportunity to see how the guy thinks.
Implementation - if the candidate is going to write code, let him do it in the interview. You will be amazed how many fail at this stage. There is a debate whether the code phase should be on paper, on a PC but only with a Text Editor (and Help?), or with a full-blown IDE. Each of the alternatives may shed light on a different interesting angle. The super-candidate should excel in all of the environments, he may be using all of them in his day-to-day life simultaneously. What would we say about a candidate that gets the IDE but when lag-behind gives the excuse that he is not used to this IDE? I believe paper exam for short implementation is good, as long as you tell the candidate that it's OK if he makes small syntax mistakes. The code should be considered as pseud-code, but all important aspects should be included (e.g. resource management and exceptions). Then of course he has to explain what he did.
Reading other people code - many developers are really good in writing their own new code but do miserably when it comes to working with other people's code. During the implementation phase you should check this with your candidate, give him some strange API that he has to use, suggest your implementation to one of the methods and ask for his opinion (it might be a good thing to throw into your implementation some obvious bug).
Effectiveness, Tidiness, Laziness - do not ignore the tidiness you see during the interview. Usually this is what you will get later. Pressure is not an excuse for a mess. Watch the way the code is placed, either on the paper or in the IDE. Laziness is OK as long as it does not effect the essence. In fact laziness is something you want to find in your candidate. You want to see that he gets things done and for that he should not finish some of the assignment you gave him, but it should be the trivial part. A candidate that cannot live with an unfinished task in a way that prevents him from even doing the parts that he could have finish, is someone that does not get things done. Effectiveness means starting with the tricky stuff leaving the trivial for the end. Identifying what is tricky and what is trivial is being smart.
How does he fit the position? - this is the place, after presenting the position and the assignments he would have to handle, to ask the candidate how does he see himself in this position, what he likes and what he doesn't, what he is strong at and what he still has to learn. This is the place to see that the candidate is attached to reality. You saw him already, you got some feelings and impression about him. Does your impression fits what he says about himself?

----------------------------------
Addition, 12/6/2008:
----------------------------------
Asking for the candidate's grades, maybe even his SAT score (Psychometric exam called in some countries) - can also be a smart thing, assisting you to pinpoint the smart guys. It doesn't necessarily help, this is why you are having the interview, but it may shed some more light. And in my experience, even if there isn't a complete correlation with performance, high grades do say something and poor grades also have their say. Usually, not a surprise, a guy with high grades would do better than his fellow with poor grades. But do not let the grades confuse you, I saw some promising candidates with marvelous grades who may barely complete an if-else.
----------------------------------

How much time should it all take?

Including the first part of attracting the candidate (speaking about the position, the organization, and yourself) - not less than 90 minutes. 2 hours may be reasonable.

__________________

Is it a true / false question?

Joel says it is. There is only Hire / No Hire, nothing in between.

I agree, if we have the budget and attraction to get all the MacGyvers that we need, then yes. I do not mean that all employees should be superheros. They should, however, be superheros in their field. If we need someone to handle XML configuration files we need someone who is a master in this domain, whistles XPaths in his sleep and grains XMLs with XSLs daily. He will not find his duty boring because he will deal with making it more efficient by inventing new utilities. Same thing would be with a secretary, we will want someone who knows how to create table of content in documents and use sophisticated attributes of the spreadsheet.

But what if we cannot attract superheros in the domain? Suppose the budget we have for developers doesn't bring us the most brilliant ones and we have to compromise.

Building your compromises

The candidate should have a positive marginal output. Writing bugs gives negative output, writing bugs half of the time may also. When saying positive marginal output, it means that you have to take into account the effort and investment to be made in this candidate if recruited.
Marginal output is related to the current mix of your team. If you have "good thinkers" but lack "coders", head for those who can code fast, even if not considered design superheros.
Remember that there are qualities that can be gained (e.g. experience), while some other qualities are inherent. Prefer smart over experienced and capable over well trained. A smart novice can quickly become smart and experienced.

Now back to the question we begun with. Suppose we recruit a Java developer. Does the candidate need to know what is "earsure" for Java Generics?

Answer is simple: if you already have in your team someone who read the Generices literature and can serve as the team knowledge base, it's not the end of the world if the candidate does not know it. The team will have the knowledge. BUT - if you are now seeking for this "curious developer", make sure he proves his curiosity!

Summary

Hiring decisions are of the most important decisions made by managers. The damage that can be made by an unfit employee are huge. Investing time in an organized recruitment process and in each of the relevant candidates is a must. At the end, the employee selected must have a positive marginal output that justifies his salary. Marginal output is related to the current mix of your team, so recruitment decisions should be made with the set of people that you have in mind, and with understanding of the qualities that you miss in your team and want to focus on.

----------------------------------
Addition, 12/6/2008:
----------------------------------

Is Smart and Gets things done the all thing? What about motivated, doesn't get bored too quickly, having a good temper?
You may want to read these comments on Joel as well.

----------------------------------

Java Generics, Erasure, Reification

2008-05-26T09:34:00.000-07:00

I gave today a talk in Sun's Java Day 2008, Israel, on Java Generics, Erasure, Subtyping, Super Type, Wildcards and a few words on Reification.

Though it's a bit of an old, Java 5, topic, and though the use of Generics is straight forward, the deeper understanding of why one gets type unsafe warnings, how to generify your own utility classes etc. - this needs some more attention. So I still find it relevant to talk about and it seems that it was relevant for most of the audience (I hope! ... well, in fact this is the feedback I got after the talk).

The presentation itself is available in pdf format here.

I want just to post here a small piece of code from the presentation. The main point of this code is how you can cast an object to T when T was actually erased and you don't have it in run-time.
The important lines are emphasized.

public <T> T getContent(Column<T> column) {
Object colContent;
String colName = column.getName();
if(colName != null) {
colContent = getContent(colName);
}
else {
Integer index = column.getIndex();
// TODO: if null throw something of your own
// (for now it will be NullPointerException)
colContent = getContent(index);
}
if(colContent == null) {
return null;
}

// now we need to cast the returned type and that's not trivial

// need raw type as a bypass
// usually use Class<?> when the type of the Class is unknown
// but it won't work here!
@SuppressWarnings("unchecked")
Class colClass = colContent.getClass();

// we know that we hold the correct type
@SuppressWarnings("unchecked")
Class<T> colClassT = colClass;

// now we can use the cast method of Class<T>
// If we were wrong regarding the type ClassCastException
// will be thrown (we can wrap with catch and replace with
// some application exception if necessary)
return colClassT.cast(colContent);
}

--------------------------------------
Addition, 27/05/2008:
--------------------------------------
A day after posting this, I got a very innocent question on the code: why not just use simple casting to T: return (T)colContent;
Well, I remembered that it's Not OK. But checking it again I recalled that Not OK means here that you get 'Type Safety: Unchecked Cast' warning, yet it does compile. What the compiler tells you is that it cannot check that the casting is fine at compile time, which is OK with us.
(Isn't it always the case that the compiler cannot check down casting at compile time? raises the question why Java decided to give a warning on this...).
So, a simpler version of the code would be of course:

// ...
// now we need to cast the returned type
// we prefer to use a local var so we can put
// the SuppressWarning here and not on the method
// (you cannot put it on a return statement)
@SuppressWarnings("unchecked")
T retval = (T)colContent;
return retval;
}

Now, one can just ask: so what's all this thing with the Class<T>.cast() method if we can directly use the "plain old" casting way of (T).
Well, in cases where Class<T>.cast() saves the 'type safety' warning, it worth using it. In our example above we get the warning anyhow (for the implementation using Class<T>.cast() we even get two), so in such a case it would be a better choice to go back to the "plain old" casting way of (T).
For an example where Class<T>.cast() indeed omits the 'type safety' warning, see item 29 in "Effective Java", details below.
Of course, if you do need the Class<T> instance itself, e.g. for reflection purposes, the original way presented above would be relevant.

------------------------------------------------------
End of Addition, 27/05/2008.
------------------------------------------------------

The idea behind the code is based on Josh Bloch's "Effective Java" 2nd Edition, item 29.

(The code example is not in the book, it's my example based on item 29. So in case of any error please find me responsible...)

Write your source as if it was open

2008-04-09T13:04:00.000-07:00

Sometimes you get to read some piece of code and you say "mmm, that's a good piece of code". Well, it doesn't necessarily occur too much, but I hope for you that it does happen, at least occasionally when reading your own code. But in many cases you come to code that makes you shout "who the hell wrote this stuff", and you may recall that it was you as well.

So, what makes a good code?

Is it comments?
Well, comments are important and we will surely dedicate a post to it sometime. But one can think of good code without too many comments. In fact in some cases comments are what makes bad code looks even worse.

Coding guidelines are surely important
Do you really think so? Making your code readable is a must, but as long as you are not doing crazy stuff you can keep your code readable without too much guidelines. Not saying that guidelines are bad, but this is not what makes good code.

Should it be simple? Should it be sophisticated?
This is in fact a target you cannot really set. The level of simplicity or complexity is an outcome of the problem you are solving. You should not try to get a simple solution for a complex problem, and you should not be too happy with a beautiful monstrosity built for a simple problem having in mind all the other problems that this pretty hideous thing may also solve in the future. So simplicity or sophistication per-se is not the target.

Lacking a straightforward answer, let's ask the opposite question...
What makes a bad code?

Bad code is a code that you cannot explain
Bad code is a code that you find hard to use
Bad code is a code that repeats itself
Bad code is a code that is tightly coupled - everything needs everything, everyone needs to know anyone else, you cannot go easily from top to bottom, there is no top and no bottom
Bad code is a code that you will not feel proud to publish under your name

To avoid writing bad code I suggest that you will think you are writing an open source. Assume someone else, that you do not have direct communication with, is going to use your code, thus your code should have clear API and be modular enough so it can be used in pieces (i.e. one doesn't need to understand the whole thing in order to create the first "Hello World"). And you should write it so you will be proud to publish it. This is writing your code as if it was an open source.

To summarize

Pay attention to your API
Modularity counts
Be proud of your code, if you are not, think why and fix it

Do you have a theory?

2008-04-09T13:01:00.000-07:00

In many cases when I sit with a novice developer who is working on fixing a bug I see him or her throwing lines of code from here to there. When I ask "why did you do that", usually when the code looks already like after a burglar paid a visit, the answer I get is "it wasn't working, so I'm trying to fix it."

Would you like someone to fix your car that way?

Debugging is considered sometimes a non-sceintific trial and error process - you try this and that, eventually the bug will be exhausted and you will break it. Well, it is not. This would only make you to mess your code and put new bugs in, break the original structure and leave the code in a strange, shaky position, usually explained later as "we needed to do that because there was a misterious bug in the original, proper code."

Just as when fixing your car, the mechanic would probably not try to shove your carburetor into the fuel tank as a possible solution (well, we haven't tried this yet, who knows maybe it works), this is not the way for debugging.

Debugging should be methodical.

Have a theory before you act
Try to prove your theory on a simple code
Divide and conquer - try to eliminate the bug by putting code in comments till the bug is gone, then analyze what is wrong in the lines commented out. But don't just throw these lines away, even if they look redundant at first sight, and don't move them around without understanding the problem
Analyze the problem and what causes the bug before you start changing your code, use logs to analyze the problem in complex scenarios
If you are changing the code as part of a theory ("I think we might have to call the super class at the beginning of the function and not at the end...") - remember what you have changed, if your theory fails (the bug persists and the change does not have a value of its own, it was just a whim of the moment) - return to the original code before you go on
If your theory prove itself right, still make sure that you fully understand it (moving the call to the super class to the beginning of the function fixed the problem - now let's understand why it should be this way... does it always have to be this way? do we need to check other similar places?)

And never try to shove your carburetor into the fuel tank, even if it makes the car go (maybe the problem was that the carburetor was too dry) - not understanding what you have done is NOT a fix!