Wicked Problems

Wicked problems in software design are problems that tempt you into a certain solution or design that seems reasonable and easy but turn out to be holey as a leach.

Mature developers and software architects know how to avoid wicked problems. Managers with lack of technical background may fall into such traps without even noticing and without realizing the trap even in retrospect, you may find a manager who would be very proud of a solution his group created for a wicked problem, investing in it a few man months, without seeing other alternatives of avoiding it or implementing it a bit differently with much less effort and side effects.

Straight solutions for wicked problems tend to be not only costly for implementing, but also for the long run, as the stitches sewed to close all holes tend to get unstitched, torn open.

Let me take you through a very small problem that I was dealing with, and turned out to be wicked.

Some background. In web applications we need occasionally to get info from the server according to user selections without turning to a new page. This is called an AJAX operation. There are several frameworks for supporting easy implementation of AJAX calls. One of which is YUI (Yahoo User Interface), which does on top of this all sort of other great things, supporting similar web experience on different web browsers.

YUI has a component for AJAX auto completion. The component is quite powerful and allows selections ranging from simple ones like this:



to more complicated, as this one:



In YUI the selection made is entered into the input-box and this is the information sent back to the server. We wanted to add the ability to send back to the server an ID instead of the selected value itself. This is useful since in many cases we want to present to the user meaningful text while the server really needs an ID. And we wanted it to work also for the multi-select version.

The solution seemed to be simply to add a hidden field and put the IDs there. The server would of course need to send in the AJAX reply the text to present in the list and input-box, as well as the ID per each entry. Making it part of the JSON reply is simple. Now, every time the user makes a selection we can add the ID into the hidden field, even for multi-select.

Simple and easy. We even found a few implementations like this on the net.

But this is not enough...

Since the user types his selection, he may type a selection without selecting (clicking on it) from the suggestion list. The user may also delete things after selecting them and then the hidden field needs to be updated. He may also retype characters into a selection made already, thus changing it to another, legal or not.

One thing came to our mind is that this feature needs to go with "Forced Selection" which means that the user cannot add things into the input-box without selecting them from the suggestion list. This may solve some of the problems. But not the deletion problem, we still need to take care of user deleting a value from the input-box. The only way to tackle the deletion problem is by populating the IDs hidden field only when the user leaves this input-box (when the focus is lost, i.e. in the "onBlur" event). Then what we can do is to loop through all values in the input-box, add a relevant ID for each in the hidden field, and display a warning for illegal values (for which there is no matching ID). The problem now is that at this point we may not have all IDs, so we need to go back to the server, in an AJAX call and ask for the IDs. And the values in the input-box must be unique, that is: each must represent exactly one ID (there cannot be two John Doe).

At this point we realized that we are going to work hard for a solution that would not bring much value to application writers, as they would need anyhow to implement a function on the server side for replacing textual values for IDs, and those textual values must be unique. So why don't just post the textual values back to the server as is, exactly as implemented originally by YUI.

Well, there is one reason for the hard work. And it is supplying warnings to the user while he types and not following post back. By checking whether all textual values has legal IDs we can warn the user for illegal values before posting the form. But then, one needs to decide whether this worth the investment, and whether it can work well.

We came to the conclusion that the people at YUI knew what they were doing. It is way too tricky to post back IDs. No, it's even more than that. It's wicked.

Now, we invested about half a day in investigating the problem and thinking of solutions, and then eventually decided to avoid this implementation. How many of us would do the same after investing a week in investigating?

A wicked problem is usually a problem that you'd better waive, avoid, abandon. Suggest other ways of doing it. Replace with another feature which is easiest to implement. The efforts of solving the wicked problem do not justify its value. Yes, a wicked problem is sexy, if you solve it you can become a true hero, it makes out a really neat feature. It is courage and experience, needed to realize the real effort for these wicked problems and to admit they do not worth it, even if some time was already invested in analysis and investigation. Or, to find the easier design, almost fulfilling the wicked problem, achieving most of the goals with much less effort.