Hibernate QuickTip: MappedSuperclass

One of the hibernate tricks for cleaner entities is the @MappedSuperclass annotation. This allows you to specify a set of fields that all of your entities share. Write a class with this annotation at the class declaration level and then make the entities inherit from this class. Without it Hibernate will throw errors.

Advertisements

Search Autocomplete Design

I’ve written half a dozen type-ahead widgets by now and definitely one of the most important aspects to type-ahead is speed. Facebook’s type-ahead is obviously really well written and might be one of the most important features on the site.

I investigated how they build their widget and one of the most interesting techniques is their use of HTML5 LocalStorage. This is one of the oldest features available in browsers, it’s only recently that it’s been standardized. There is a cool library you can use to abstract the browser quirks. Here is summation of techniques that I’ve picked up:

  • Preload. First page load, grab a finite set of probably searched entities. In Facebook’s world it means grab all your friends or top X friends if you have a lot.  Cache this in Local Store. You’ve now covered 80% of the cases.
  • Show results right away. Chances are the user will see something they are looking for already, and if not while they are scanning the possible list of entities you can fetch the long tail list of entities they could be looking for and drop them in.
  • Filter in memory. Some type-ahead widgets will filter on the DOM. Terrible idea because DOM manipulation will always be slow. Filter in memory with some regex magic.
  • Optimize your Regex. Regex is an art. Rely on someone who can write you a good matcher. Or ask me!
  • Memoize in memory. Chances are they will hit backspace. You want to save every mutable keypress.
  • Use LocalStorage aggressively. People search for the same stuff over and over. Save the bandwidth and time. Cache the search queries. This is where LocalStorage comes in handy!

Happy coding.

OuterHeight in Prototype

Jquery comes with some sweet utils that prototype users miss out on sometimes. One of my recent discoveries is outerHeight. Prototype comes with Element.getHeight() method, which only gives you the actual height of the element discounting the margins, borders, and padding. Outerheight in jquery gives you the whole thing. Here is the implementation for Prototype:

Element.addMethods({
	outerHeight: function(element) {
		var height = Number(element.getStyle('height').slice(0,-2));
    	var padding = Number(element.getStyle('padding-top').slice(0,-2)) + Number(element.getStyle('padding-bottom').slice(0,-2));
    	var margin = Number(element.getStyle('margin-top').slice(0,-2)) + Number(element.getStyle('margin-bottom').slice(0,-2));
    	var border = Number(element.getStyle('border-top').slice(0,-2)) + Number(element.getStyle('border-bottom').slice(0,-2));
    	return height + padding + margin + border;
	}
});

Now you can get the height of any element you want with just $("mydiv").outerHeight();

How to Fix the Android App Store


Let’s face it the iPhone app store is more popular and more profitable than the Android app store. In recent reports 65% of the Android app store is free, while only 30% of the iPhone app store is free. It begs the question of why companies and developers are avoiding Android.

It could really be many facts, but here are some that are evident:

  1. It’s really a chicken and egg problem: either the app store doesn’t have any quality apps and no one wants to buy them, or because no one buys apps on Android no one wants to build quality apps.
  2. There is no quality control on the Android unlike the iPhone where every application is filtered and approved by Apple.
  3. Android is used as a cost cutting OS, therefore being put on cheaper phones where people are less likely to purchase applications.
  4. Apple came first. A lot of companies have already invested in Apple and it seems to be profitable, why build for another platform.

Google can’t really change most of these, but the first two. The second is problematic since it is an open platform, but I argue that the first is actually their biggest problem that they need to tackle and they are in luck since that one can actually be solved.

The solution is simple. Apple and Amazon are already doing this and very successfully. People need to be comfortable with the idea of buying applications. What Google can do is remove the barrier of thinking about your purchasing decision. Make it one click. Apple/Amazon don’t accidentally collect your credit card and enable one click shopping. How does Google do it? Here it goes:

  1. Any android has to be registered with a Google account.
  2. Offer users $5-15 of app store credits when they associate an account with a credit card. Make the credit subsidized by the cost of the phone.
  3. Enable one click buying.

This solves the problem #1 from above. Once more apps are bought on the Android, companies will flock to the platform. This than makes the Android app store more profitable.

Why Apple TV Fails and What Apple Can Do About It

One of the biggest announcement today was the new Apple TV. The giant is continuing it’s model of making their devices smaller, thinner, and more colorful(I call it the  iPod model). However, this device is not portable and although it looks “cooler” it is by no means any more functional or useful. The reception is lackluster just like when the original Apple TV was announced.

The iPod model doesn’t work for TV smart boxes. When the iPod came around it was a complete replacement for your music device. No one used CDs anymore, or record players at least for the most part. This is why this device is no game changer. I still have my blu-ray player because it isn’t the same quality, and I still have my cable because it isn’t the same breadth of content. It doesn’t win.

What can Apple do about it? I think the real innovation is not in the device, it’s in the pricing model. I wont purchase a show on Apple TV every day unless I missed one and really wanted to watch it. How can you get people to purchase couple of shows every day? By abstracting the purchasing decision. What Apple needs is a subscription model for TV content. This way they replace your cable, and they truly own your TV. This removes the user from the penny gap, the decision to watch something becomes unconscious.

How do you sell something like this? I think it is easy. Comcast charges you at least $100/month with 90% of the content you do not watch. Apple can charge $60  and  it’s exactly what you want to see and when you want to see it. $60 dollars buys you 60 episodes — that’s 2 a day. Some people will watch less(big profit), and some more(small profit), but the margins would be significantly more than what Apple would be making just selling the device.

Javascript Error Logging Infrastructure

If you are running a large scale dynamic site than you are familiar with logging on the server side. This immediately notifies you if there are any issues with your JSPs/PHP, broken controller code, data integrity, etc. Common practice is to log everything, accumulate logs, and generate automatic reports. Most errors will bubble up within minutes of a code push.

One problem with this is that no one javascript into the picture. Javascript errors are very evasive due to the wide variance of browsers on the web. QA process alone will likely catch backend errors more often than front end errors since the backend environment is static while the front end is not.

I propose something very simple and elegant. The requirements are few:

  • System must tie into the already used backend logging infrastructure
  • The processing and code length must have a small footprint as not to effect performance.
  • Captured information must be meaningful: need to know where it happens, why it happens, and how it happens

Here is sample code using Prototype JS:

			var newHandler = 
				function(msg,url,line) { 
					new Ajax.Request("/error", { parameters: { "msg": msg, "url": url, "line": line, "browser": navigator.userAgent } });
				}; 
			var oldHandler = window.onerror;
			
			window.onerror = function(msg,url,line) {
				if(oldHandler) { oldHandler(msg,url,line); } 
				newHandler(msg,url,line); 
			}; 

With some Spring Controller handling:

@Controller
@RequestMapping("/error")
public class JSLoggerController {
	static final Logger logger = Logger.getLogger(JSLoggerController.class.getName());
	@RequestMapping(method = RequestMethod.POST)
	public String onSubmit(@ModelAttribute("message") String msg, @ModelAttribute("url") String url, @ModelAttribute("browser") String browser, @ModelAttribute("line") Integer line) {
		logger.error(url+" | " + browser + "\n" + msg);
		return "redirect:pixel.png";
	}
}

In production environments this can be rewritten to be a batch API. Notice that we capture the browser information as well, but we do not process it so that there is lower overhead. The server can download the file and pull the lines before and after the error to be even more expressive. The server piece here is trivial: we bind the error URL to a Log4J(or other) logger and print into the logs we are already processing.

Firebug console Object Degradation

If you are using firebug’s console object to print messages to firebug this will break functionality for clients that are not using firebug(which is everyone). Here is a small bit that I found on Firebug Lite that solves this problem:

if (! ("console" in window) || !("firebug" in console)) {
    var names = ["log", "debug", "info", "warn", "error", "assert", "dir", "dirxml", "group"
                 , "groupEnd", "time", "timeEnd", "count", "trace", "profile", "profileEnd"];
    window.console = {};
    for (var i = 0; i <names.length; ++i) window.console[names[i]] = function() {};
}

Front End Performance Lessons from Rotten Tomatoes: Caching

I’ve been spending a lot of time obsessing over making things faster at rottentomatoes.com. In this multi-part performance series I’ll go over various thing’s I’ve tried and which of those things were successful. Some of them are evangelized by Steve Sauders, and some not mentioned at all. I want to first focus on caching. When it comes to the browser there are various possible caching levels:

  1. Data caching: this is caching at the data layers(db), this is very commonly accomplished with distributed caches like memcache and/or in memory cache like ehcache. If you have a site that has at least a million users you are already doing this. This is a well covered topic, so I will not be discussing it in this post.
  2. Dynamic content caching: regenerating dynamic content is expensive. One JSP page can hit the database/memcache many times in one request. Consolidating all the generated content into one memcache call can possibly yield faster response times.
  3. HTTP caching: often times overlooked part of speeding up performance. HTTP came built with an elegant caching solution. By setting ETag and cache-control headers for content that doesn’t change often you can reduce server load and improve page performance. The HTML content is not loaded again, saving on bandwidth and response latency; plus most browsers are optimized to display the page faster  if nothing has changed.

JSP Caching

We are obviously already doing #3 at Rotten Tomatoes. So the first caching mechanism I went after was JSP caching; the main focus of this post. The most sound way of implementing this is by creating a custom tag. This way the author of a particular page can choose to wrap the content in a cache tag and future modifiers are aware of what parts of the page are being cached. Some content can be very dynamic and so you want to give granular control to the developer so they can make their own decisions. To sum up our requirements:

  • Granular. Allows from sections to the entire page to be cached. This allows common shareable sections to be
  • Visible. Developers have to be aware where it’s being employed.
  • Simple. Should be easy to drop in without having to consider what’s happening behind the scenes. Should assume safe defaults (caching for a couple hour is a bad idea).
  • Transparent. You should know when/what is cached and be able to see the page without the cache.
  • Fast. Obvious goal at the end of the day is that we shouldn’t be hitting the cache multiple times at the view layer per request.

The defaults I chose were to put all the JPS cached content into their own regions keyed off by TTL. The default TTL being 15 minutes, and the default key being the request URL. Both values can safely be overridden and usually are depending on the context they are being used. Setting up the tag is easy:

public class JSPCacheTag extends SimpleTagSupport {
    public void doTag() {
        String cached = memcache.get(key, ttl);
        if(cached == null) {            
        StringWriter buff = new StringWriter();
            getJspBody().invoke(buff);
            cached = buffer.toString();
            memcache.put(key,cached, ttl);
        }
        getJspContext().getOut().write(cached);
     }
...
}

Due to the nature of JSTL tags we have already succeeded in all of the requirements that we set to accomplish. The one remaining problem is nested caching.

Nested caching

One problem that we encounter with this approach is that if we cache the entire page, and children of that page are also marked to be cached than we will be violating our simplicity requirement and speed requirement. Every time the entire page is cached, we will then be checking and possibly caching children elements as well, which would be unnecessary since the entire page is cached. There is a way to avoid this. You can set a boolean request variable that is set when the parent is cached; the children will check for the variable and if it exists will not cache their content. This is of course a micro-optimization for reduced latency and to reduce the use of your cache space.

Result

For rottentomatoes.com the effect of the JSP cache was significant. On average the JSP generation step went from ~400ms to ~20ms yielding a noticiable reduction in request latency.

HTTP Caching

HTTP caching is often highly leveraged for CDN bound static content, but unused for dynamic content. Setting the cache is best done in a request Filter by setting the headers. The implementation is definetly simpler than the one for JSP cache. Sample pseudo-code:

response.setHeader ("Cache-Control", "max-age=86000, must-revalidate");
response.setHeader("ETag", MD5Hash(content));
if(request.getHeader("ETag").equals(MD5Hash(content)) {
    r.setStatus(304);
}

This is one of the next steps in our caching optimizations. Early testing gives very positive results.

Conclusion

Hopefully this has give you so me ideas on how to leverage caching in different ways for increased front end performance in your project. These techniques are mostly helpful in a semi-static environments. In cases where the content is highly dynamic and user specific one you can leverage HTML5 local storage, but that is a topic for another day.

Color is King. Color Picker for CSS.

Color is fun, but what’s not fun is when you need some nice color for a design and you can’t find it. I found a cool little site: http://colorschemedesigner.com. The color wheel shows a wide variety of gradients and allows you to choose like and complementary colors. Great feature that this tool has is to preview all the chosen colors on a site. Only improvement I would make is to copy and paste the color code into the clipboard so I don’t need to use firebug to grab it.

MySQL adding foreign keys doesn’t work

I ran into an interesting problem recently where the foreign key constraint on my MySQL table wouldn’t stick. Consider the simple case:


create table duck (
id bigint,
name varchar(10),
parent_duck bigint,
foreign key(parent_duck) references duck(id) on update cascade
)

Now if you do a show create table. The foreign key constraint is nowhere to be found. What’s the deal? Dig around on MySQL documentatoin and you finally run into(http://dev.mysql.com/doc/refman/5.1/en/ansi-diff-foreign-keys.html)

For storage engines other than InnoDB, MySQL Server parses the FOREIGN KEY syntax in CREATE TABLE statements, but does not use or store it. In the future, the implementation will be extended to store this information in the table specification file so that it may be retrieved by mysqldump and ODBC. At a later stage, foreign key constraints will be implemented for MyISAM tables as well.

So the way to fix it to declare the table as a InnoDB: alter table duck engine=InnoDB;

Why isn’t InnoDB default and why is MySQL silently ignoring the constraints is anyones guess :).