NFC vs Acoustics, Mobile Payments and HopOn

NFC (Near Field Communication) is an industry standard for communication between mobile devices as well as between mobile device and peripheral hardware, for tasks as mobile payments or controlling mobile functionality by area, or by proximity to some other device (see Samsung TecTile sticker for a cool use case by Samsung).
Note that NFC and Bluetooth are not the same technology (see the difference between the two here).

While it seems as a promising standard, adopted by most new mobile devices, iPhone still doesn't. Which makes it a bit difficult if you want to implement a generic application with communication between devices, or between device and hardware, and the device might be iPhone. Applications in that domain might be games (e.g. a bingo game that invites all devices in the room to participate, without a need to "register" or "know" the arranging server domain), broadcast communication in a closed environment (bus, train, airplane) and of course mobile payment.

As an alternative to NFC, some companies have proposed the interesting idea of using un-hearable, ultrasound, sound waves communication, companies like Brazilian startup NearBytes, Microsoft and others more. Idea is to use modern mobile device abilities to receive and to play inaudible signal (>18kHz), supported for example by iOS and Android.
(Read more about NearBytes here).

I came to this recently while following Israeli startup HopOn, presenting a working system for transportation mobile payment based on acoustic signal broadcasted by a small hardware device on the bus. Their technology seems promising and is already working in some bus lines in Tel-Aviv. See their marketing piece here, it's in Hebrew but you don't really need to understand Hebrew to follow it.

Big Map, String, GC, Memory Consumption and Performance

Hashtables are used in many cases to cache information in memory. There are other alternatives, like in memory databases, or flushing data to disk, but when all you need is a simple key-value relation and you need very high performance, the simple Java HashMap can do the trick. Usually cache can allow loosing information that was not claimed recently, assuming that recent usage of the data can foresee higher chances of this data to be claimed again soon, compared to less recently used data. But in some cases we prefer the cache to hold as much information as possible, which means a VERY BIG MAP.

Important morals below are based on an application with HashMap holding 20M entries, in 64bit JVM and 6 GB (-d64 -Xmx6g ). The Map itself uses much less memory, but in order for the entire application to work smoothly we need the 6 GB.

When coming to very big HashMap, there are a few considerations to take into account:

1. The default load factor of 0.75 is reasonable. Don't play with it without understanding what you are doing. The load factor determines when the map should grow and rehash should occur, this happens when number of elements in the map gets above the result of (capacity * load factor). Higher values of load factor means less rehash operations, but a map that is more dense. Calls to get in a dense map are more costly since buckets are full with many entries that shall be iterated. Lower load factor values will create more rehash operations and a sparse map, which is more memory consuming but more efficient in get calls. It should be noticed that load factor can be bigger than 1, which means we allow the table (number of buckets) to be smaller than number of entries.
   
   To summarize: don't start with optimizing the load factor, it's a bad start.
   


2. Initial capacity is important. Starting the HashMap too small will result with rehash operations upon calls to put. Though HashMap doubles itself on resize, still failing to provide appropriate initial capacity may result with a few costly resize cycles.
   The recommended initial capacity, for load factor of 0.75, is: 1.5 * numElements
   (Read more: http://stackoverflow.com/questions/434989/hashmap-intialization-parameters-load-initialcapacity).
   
   In our case, the performance difference of setting 20M entries into the map, with setting the initial capacity to 30M (Java will in fact set the initial capacity to the closest power of 2 above this size), compared to no initial capacity set, was almost double the insertion time, with very clear workload on rehash and GC work following it.
   
   To summarize: initial capacity is important. Set it!
   


3. Beware of memory leaks!
   Maps, as any long-leaving memory storage entity, are a potential source for leaks.
   In our case, the Map got Strings from file, but needed to use for the entries' keys only a small portion of each line read from the file. Thus, after reading each line from the file, some string tokenizing was done resulting with the key and value to store.
   Unfortunately the entire line read from file is kept in memory, even though we need only part of it... This is due to the way substring and other similar tokenizing methods are implemented, which return a new String, that holds the old one with offsets inside. This is a known issue (for example see: http://eyalsch.wordpress.com/2009/10/27/stringleaks/).
   Solution is to manually create a new String and give it the substring portion you want to keep:
   String key = new String(bigLine.substring(from, to));
   In our case, fixing a similar issue resulted with requiring 1.5GB less memory for the exact same scenario! It turned out that the map keys' kept a long tail that should have been cut.
   
   To summarize: beware of holding a substring in a Map! Do it the right way.
   


4. In such a huge map, every byte in the key-value becomes 20M in the entire map. Thus saving a few bytes can worth the hassle. For example, instead of using String as the value, we decided to use char array (a simple char[]) thus saving: [1] a reference to the string object from the hashmap entry - instead holding a direct reference to char[]. In 64 bit this values to 8 bytes saving! [2] three int fields that are in String class: offset, count and hashmap. The offset and count fields are there to allow String to point at a position which is partial in the char[] it holds (directly related to the substring leak mention in item 3 above). We don't need this info. Also caching the hashmap value is redundant, as the Map entry itself does it.
   
   By using char[] instead of String as our value, we save 20 bytes per entry, which is 400MB total! And without hurting the code or making it more complex.
   
   Our key is also a String, but it cannot be turned into char[], as we need a proper hashcode and equals functions. Still, one should consider implementing a dedicated Key class that holds only the relevant info, char[] in our case. Since the extra reference cannot be saved, the saving here could be the 3 int fields in class String, which totals to 240MB for 20M entries.
   
   To summarize: caching Strings in big Maps is costly. Think how to reduce the size of your key and value!
   


5. The hashcode is crucial. Bad hashcode on the key, which creates bad distribution (too many keys get the same hashcode value) can be performance devastating. On the other hand, though of much less importance, a more efficient hashcode method has an influence over at least 20M insertion operations. Read how String hashcode is implemented to get the idea: http://stackoverflow.com/questions/299304/why-does-javas-hashcode-in-string-use-31-as-a-multiplier
   
   To summarize: check your key hashcode method, use some real data and validate that you get reasonable distribution. If needed, work on improving the hashcode method by using higher power multiplication on the key fields.
   


6. Know when to stop optimizing. In some cases you see a programmer keep optimizing when there is really nothing more to optimize (well, there is always more to optimize, but then it may result with taking parts to C and using JNI, or use direct byte arrays, things you would better not go to, if not really needed...).
   Calculate the most minimal amount of memory you need, save your time and don't try to optimize your application into less.
   
   To summarize: set goals to your optimization efforts. Make sure the goals are not too aggressive, i.e. are feasible without re-writing the entire thing.
   


7. The entries of the big map will eventually arrive to the GC old-generation section. It means that in the described usage the old generation is going to be very big. There are two options to handle this, one is to play with the generation ratios, the other option is to set a big enough total heap memory to allow the old generation to be big enough. If you have enough resources on the machine you can go with the second option, otherwise you would need to dig into the generation ratio configuration.
   
   To summarize: size of the old generation section is crucial to prevent unnecessary full GC cycles. Configure the max heap size and/or the generation ratios.
   

By optimizing the memory consumtion we got also much better performance, as the application went into less GC cycles. Some other parts of the drill include removing unused entries from the cache, in a dedicated thread, when reaching certain threshold, and more... The result is a huge cache running smoothly on a reasonable sized server.

Thoughts following 2010 FogBugz and Kiln World Tour

I was attending yesterday Joel Spolsky world tour for FogBugz and Kiln.

I must admit that till yesterday I thought that Joel has a great blog about SW development, but I didn't quite understand - who needs yet another bug tracking tool... is there still a real market for that, when you have today so many good free tools!?

A few thoughts following the event:

1. Joel's presentation skills do not fall from his writing. The gathering starting with a projected countdown analog clock, Joel went on stage right at the moment when the countdown reached its end - with a pre-planned bug on the clock finish resulting with some fake errors and blue screens, which led Joel to open a bug, start his presentation, then get a response on his newly opened bug "poping accidentally" while in his presentation (interestingly enough, while his Outlook is closed :-) , which led to getting into the code itself using Kiln, comparing versions and eventually "solving" the bug and checking in. A full bug detection and solving cycle in 10 minutes. With a few jokes here and there, and while going quickly through the products' abilities and strength points.
   
   


2. The traditional tools for bug tracking and source control are OK. But even in this "already solved" domain, there is still room for improvements and for new players, either open source or commercial. When we have an idea to develop something we usually tend to check who already done that and how good it is. And when we see that there are already several reasonable solutions we assume that the market is closed for us on that. Well, Joel shows that there is always a market for realy good products, you don't have to invent a new big thing, you may need however to invent many small things.
   
   


3. Distributed Source Version Control - GIT, Mercurial, Kiln - disconnects the developers own copy from a repository, while preserving full history and repository context. This concept has several significant advantages:
   (a) You do not postpone your check-ins, being afraid of hurting the main repository, check-ins are made into your private copy. When you are ready you push your copy back to the main repository. Your check-ins preserve full history during your development! This is also important when you perform a merge and realize that your original file was overriden by a wrong change - you don't have to worry as you have your full history at hand and you can get back to your last check-in.
   (b) You can push your developments to another repository, e.g. to a QA repository, thus allowing to push relevant fixes quickly, without releasing them yet to the development repository.
   (c) Upon merge, it's easier to see for each change the exact origin of it. Managing several customer releases you can see which changes in the main release where merged into the customer release and which not.

Online IDE for almost any SW lang you can think of

Take a look at this one: http://ideone.com/


It supports:
Ada, Assembler, AWK, Bash, bc, Brainf**k, C, C#, C++ se, C++0x, C99 strict, CLIPS, Clojure, COBOL, COBOL 85, Common Lisp (clisp), D (dmd), Erlang, F#, Factor, Falcon, Forth, Fortran, Go, Groovy, Haskell, Icon, Intercal, Java, JavaScript (rhino), JavaScript (spidermonkey), Lua, Nemerle, Nice, Nimrod, Objective-C, Ocaml, Oz, Pascal (fpc), Pascal (gpc), Perl, Perl 6, PHP, Pike, Prolog (gnu), Prolog (swi), Python, Python 3, R, Ruby, Scala, Scheme (guile), Smalltalk, SQL, Tcl, Text, Unlambda, Visual Basic .NET, Whitespace


It's very useful when you want to check little programs, like the ones I tried when writing Freaking behavior of a small little C/C++ bug:

[1]


[2]

Thoughts on Open Source licenses, Patents on Software and such

I'm dealing with Open Source usage approval cycle, which is an important task in any company, last thing that you want is to have your developers use whatever they find on the web.

Few insights and thoughts from my experience:

• Developers in genral are mostly ignorant to legal issues. If not controlled they may use a free 30-days evaluation copy embedded in their system, just because the word free appeared somewhere in the site. In most cases they don't bother to read the license.
  
• With Off-Shore developers problem is even bigger. They tend to be much more open with open source, without seeing the risks. Even if they do follow the company policy, submitting usage requests for open source usage, you may find inside their code much more "embedded" un-approved snippets and libraries. I tend to think that the reason is the distance, they believe that even if caught the maximum you could do is yell at them over the phone or in e-mails, but you cannot beat them physically and they use it.
  
• For above reasons and others, usage of open source must be controlled. There are scanning tools in the market that help you find un-reported usage of open source and commercial external software. Usage of such is helpful in finding the disobedient developers who still drop in whatever they like, fix that on time and beat them while the felony is still hot.
  
• Scanning tools also point at many usages that are a very small snippet of something that looks like might be taken from an open source or even from an un-licensed example on the web. To some it seem a problem that should be fixed in the code, I personally believe that the rights on how to perform quick sort do not belong to anybody, even if part of some open source or are published on the web somewhere. Taking two notes from a melody doesn't harm its rights.
  
• Same goes for patents on software. Publishing a patent on algorithm is problematic, but many patents are on "a method and a system". I have such one myself. Does it really prevent anyone from creating a new similar development? Should it?

Java Unit Testing

There are a lot of tools and options out there for java unit testing, replacing JUnit or on-top of JUnit. Let's do some order in things.

1. TestNG vs. JUnit4

TestNG came out to add features that were missing in JUnit 3.x.
It did quite a good job and some may say that TestNG 5.10+ is still better than JUnit 4.7+
TestNG has better data providing mechanism.
TestNG has test dependency definition, missing in JUnit.
TestNG has group level with fixtures on group. And it also has the ability to create automatic run file for all failed tests.
All that said, JUnit has more extending libraries using its ability to extend JUnit TestCase and JUnit TestRunner.
Both have very good IDE support, as well as Ant and Maven support.

Coming to choose, both are good. I stick with JUnit.

Note that JUnit 4 made the mistake of integrating another library into its jar, that is the hamcrest core. They should have known better than that... The problem is that when you get the full hamcrest lib (to get the really powerful matchers that you need in your tests) it gets confused with the lib that came with JUnit. Solution is simple: put the full hamcrest that you bring to be first in the classpath, before the JUnit jar.


2. Load

JUnitPerf do the job with its ability to run tests in several threads.
It's easy and useful, for integration stage and for any piece of code that may act differently under load.

The following code was tested with JUnitPerf and it catches the bug of not synchronizing the addIfAbsent:

public class MyList extends ArrayList
// we test with the synchronized and without
public synchronized boolean addIfAbsent(Object o)
boolean absent = !super.contains(o);if(absent) {
super.add(o);
}
return absent;
}
}


The test that catches the bug when the synchronized is deleted:

public class MyListTest extends TestCase {
private MyList myList = new MyList();
private int count = 0;
MyListTest(String name) {
super(name);
}

TestSuite suite = new TestSuite();
Test testToRun = new MyListTest("testMyList");
int numUsers = 1000;
int iterations = 5;
suite.addTest(new LoadTest(testToRun, numUsers, iterations));
return suite;
}
public void testMyList() {
if(myList.addIfAbsent(count)) {
// there is a gap here that may cause error on correct
// implementation, in case of thread switch at this point
synchronized (this) {
++count;
assertEquals(myList.size(), count);
}
}
}
}

This solution is not bullet proof.
Theoretically it may give errors on correct scenario, and of course it can always miss bad implementation, as it’s a matter of timing.
However, in reality it does catch the problem!
JUnitPerf is simple and gets the job done.


3. Thread testing

It could be nice if we could have time the behavior of the code under test, to mimic race conditions and check how the code operates. The idea is to deliberately create race condition then to check that a synchronization lock is working as expected, ensure that unlocked blocks are fine, check for deadlocks and see if there data corruption.

With such ability we could have create the race condition in the above code with a deterministic approach instead of using load.

Suggested tools:

• thread-weaver (version 0.1)
  allows the test to time different threads reach code points in a certain order, either explicitly or “semi-automatically”
  
• MultiThreadedTC (version 1.01)
  allows to set time tickers on objects in the test itself, but NOT on the tested code, thus less relevant for most testing purposes

Both tools are not highly supported, recommending still not to rush into it…

4. Mock Objects

There are 3 cases where you need a Mock Object in your test:

(a) Tested code gets a complicated object as parameter
Solution: Mock the parameter with a Stub/Mock implementation

(b) Tested code invokes a static call on some resource
Solution: Replace the static call with a Stub/Mock implementation

(c) Tested code creates complicated objects
Solution: Replace the created objects with a Stub/Mock implementation

The Mock utilities rely on one of the following technologies: java.reflection.Proxy, replacing the ClassLoader, byte code instrumentation and AOP weaving (which relies by itself on replacing the ClassLoader or on byte code instrumentation).

Possible tools: Mockito, JMock, JMockit, EasyMock, JEasyTest and many others…

There are differences between the tools in syntax and abilities.
Tools that rely on Proxy cannot mock static behavior and object creation.

The two possibile combinations that came to my final round were:
(a) JMockit 0.998
(b) EasyMock 3.0 + PowerMock .1.3.8

JMockit seems powerful and well documented, but in one of my tests the test failed without a reason and when I debugged it I saw that some exception is thrown within the JMockit code itself. Probably I did something wrong in the test, but still this is not what I expect for. Maybe the 1.0 version would be better...

My selection here would be EasyMock + PowerMock


5. Other tools

Other tools in this domain worth mentioning:
- DBUnit
- HttpUnit (not only for tests)
- Selenium (it is a good tool for Web UI tests in general, and you can operate it from your java unit tests if you'd like)

Technology Optimism

At the time of massive cutoffs, transition of positions to low-cost centers and the sour smell of pessimism in the air -- I feel the spirit of Technology Optimism.

During the good times, the finest people with the brilliant ideas are sucked into the big corporations, tempted by nice salaries and all other perks. In many cases what they do in such positions is to work hard on projects that are later being shutdown. Let's admit, the big corporations are good at raising the money and selling the product, but when it comes to innovation, best innovative trick is to find the right startup to acquire. And this goes also for great innovative companies like google (GoogleMaps, GoogleVoice and many more are based on external acquisitions, see Wikipedia - List of acquisitions by Google).

There is a great advantage in a crisis, being a tool in reforming the market back to an efficient structure. We need more innovation and much more efficient companies.

Many brilliant people who have been back into the market lately are fed up working for giant dinosaurs, and start working on their own new startups. It may be hard at this point to raise the initial capital, but these startups are thin and efficient. And in a year or two we will start seeing the results with new technology domains that we couldn't imagine before.

IE is a pain in the ass

Back in the old days when there were Internet Explorer (IE) and Netscape, everybody knew that it is very difficult to write something that would run perfectly on both. But since it was only two dominent browsers not many pointed the finger at IE. It was just that the two are different, life is challenging what could you say.

Today, when you have Opera and Chrome and Safari and FireFox -- ALL BEHAVING THE SAME -- and IE in his own realm, it is much easier to point the finger.

And here are some who do point the finger:

http://whyiesucks.blogspot.com/

http://iesucks.net/

http://zipthefuckup.com/2009/03/31/ie-is-a-pain-in-my-ass/

http://empireenterprises.com/noie.html

http://www.techsupportforum.com/design-forum/web-design-programming/111095-css-firefox-good-ie-bad.html

http://channel9.msdn.com/forums/Coffeehouse/427835-Why-is-IE-so-bad/

and there are more, just google for IE sucks or IE pain in the ass. Which it is, for most web developers.

It's nice to see that the other browsers do have a strict implementation according to the W3C standards, resulting with uniform behavior and ease of development. It says good things on the W3C standards being well defined and about the discipline of the coders of the other browsers, as opposed to IE.

Microsoft needs to drop IE altogether, start with a new code base and a new brand name, the current one is ruined.

JavaDay, JavaOne - Java Thoughts

I was attending Sun's Java Day this week, here in Israel. The main themes Sun is pushing are JavaFX and MySQL.

Some thoughts on both, as well as other topics.

JavaFX

Sun's interest here is clear. Every handset or any sort of device running JavaFX means more licensing fees for Sun. In fact, JavaME is one of the only Java environments which Sun's really see money from, at least for now. This would probably change in the future with much more agressive sell of JavaSE licenses for extra features and support, but still handsets is a very important addressable market, competing with others technologies such as the simple browser, Android, iPhone Objective-C, etc. Sun interest is to have as many applications as possible running on top of JavaME, JavaFX and LWUIT (the last two are a bit competing on the same square). More applications means more consumer power (consumers who want to run these applications) thus more pressure on the handsets manufectures to have JavaME and JavaFX on the handset, droping the required license fee at Sun's pocket.

All fine till this point. But now it comes to JavaDay event, or JavaOne conference in SF. These events shall reflect what interests the Java community, be judgmental, compare competing technologies. But instead it becomes more and more a marketing event for what Sun is trying to push. Thus, JavaFX catch very high percents of all sessions (even a session on Maven had to be called "Deploying JavaFX apps with Maven" or something alike to get in...) And it is totally lacking the critic perspective, comparing with other competing technologies such as Android, GWT, zool, YUI, Adobe AIR etc.

Is JavaFX interesting? probably yes, but not at the amounts poured at us.

As for the judgmental perspective, I'm missing in JavaFX the separation between content and style, without that this would be a step back. Does JavaFX work with CSS? not that I know of. Is there a similar alternative? The declarative approach of JavaFX is good, but why the choice of JSON-like declarations and not XHTML? The language is very close to Groovy, but not exactly the same (Groovy is from Spring creators), is there a path to consolidate these two new scripting languages? And why should we go with a propraietary environment and language, and not with W3C standards adopted by the browsers running on mostly all handsets? ... Yes, I know there are answers for some of the questions above, but in the huge time dedicated for JavaFX, alternatives, as well as known flaws and drawbacks, were not discussed.

Which brings me to the thought that independent Java events, such as Devoxx, are a better meeting place for the community.


MySQL

This was another theme at the event. MySQL is a powerful tool, the regulator shall take it out of the hands of Oracle to preserve competition.


Java Reference App

Together with Ronen Perez, I was presenting a session called "Java Reference Application - putting all the buzzwords together" focusing on our selected libraries and tools.


Java Community and New Technologies

It's good to see all these smart people together. Being smart, people follow the trends and new technologies carefully, asking "what is it for me? how does it make my life easier?" In the past you could see people enthusiastic about any new thing, today you see people much more careful and reluctant. It's either that we are fed up with disappointments from previous "innovations" or that the pace of things require us all to be more cautious and conservative. This, together with the competition between technologies, requires new technologies to be well cooked and invest much more effort in getting us in.

Technological Singularity

A singular point is a point in which the rules that define the spectrum around are no longer valid. In Mathematics a singular point is the point for which a function is undefined, in Physics, Gravitational Singularity means infinite gravitational field, known also to be a Black Hole.

Technological Singularity is the point of time when computers would be more inteligent than humans, thus creation of new computers and software programs can better be made by computers. Though sounds science fiction, something like the Matrix, I do know, as of today, of humans that are less inteligent than computers, one shall admit computers did part of the way.

The steps towards singularity in software engineering would be the creation of new software languages which are much more declarative rather than procedural. Computers would be able to get well-formed requirements, or constraints, to create the required piece of software. Prof. David Harel started talking about it back in 2000. Now the problem with our usual requirements is that they are not so well-formed, filled with holes and hacked with contradicting constraints. But what if a computer can identify the holes and contradictions and ask us what we really meant, then go and create the software. And maybe there is no need to ask us, since the computer knows better anyhow. The famous inventor and futurists Ray Kurzweil is speaking about technological singularity in his book The Singularity is Near with forecasts such as that by 2020 personal computers will have the same processing power as human brains and by 2030s Mind Uploading would become possible.

It's closer than you think. Declarative programming. Computers who understand requirements. Technological singularity. At which point the only occupation still relevant for humans would be lawyers, as computers cannot be mean enough for this job.

The future is surely an adventure. Some would see the upside potential of it, others may see the risks, which are there of course. For those and for those it would be good to read some advice from Daniel H. Wilson, on How To Survive a Robot Uprising: Tips on Defending Yourself Against the Coming Rebellion.

Google open sourcing their new browser, Chrome

Google new browser, Chrome, is the talk of the day. And the comics are indeed great.

Some were enthusiastic about Google opening this vast project as an open source. Noble indeed. Google themselves are feeling virtuous about it, or at least want us to feel that way. After all, a great tool like this (no one really seen it yet, but they do know how to create a buzz and I guess it will be indeed a good browser) – by releasing this great, superb, new-era browser as an open source, anyone can take it and adapt it to his needs, even Microsoft can, and maybe will even do.

So why did they do it? Why to open source? You can give it for free without opening it...

Slide 37 lists the noble arguments.

But could they really?
Do they have the option the ship a proprietary browser?

NO, because they don’t have one.

This new Chrome browser is said to be built upon WebKit and Mozilla projects. Both projects are open sourced and require derived work to be open as well (section 3.2 in Mozilla Public License and 2 of the LGPL).

They could have find a way to keep some of the code closed, like maybe the V8 javascript VM. But it would probably be too risky legally. And they might need their lawyers ready for Android issues (or for buying Sun).

Bottom line:

No need to be too cocky about open sourcing. As nice as it is that you open it, you probably had no other choice, other open source initiatives worked hard on writing parts of your code.