Haystack is the Facebook fileserver for photos. Instead of using costly NFS operations (as they did in the past) they adopted a simple log based file where the typical operation is an append to an already open file. Obviously they have a separate index structure, and many cache server in front of the system.
This is a typical solution when you have a lot of new objects inserted in the system (and a limited amount of delete operation). If you have many delete, you probably should consider something like the COSS (Cyclic Object Storage System) developed for Squid.
This is a nice example of building something with the minimum set of features necessary to get the job done. Sometimes less is more.
ReplyDeleteThis is a typical solution for a search engine when you need to serve documents. DocStore is typically a log based append store.
ReplyDeleteBTW, this is presentation for Haystack http://www.facebook.com/video/video.php?v=572283147938&ref=mf check it out since it is quite interesting
That doesn't change my point, Antonio. Facebook has a pattern of building the minimum set of features necessary to solve the problem at hand. Microsoft could learn from that.
ReplyDelete