Infrastructure for Modern Web Sites

One of the things that I did when I wrapping up at Yahoo! was to begin to take a look at the current state of web frameworks. I ended up picking Django, but I have to say, I was disappointed with the state of what’s out there. Friends will have heard me bemoaning this sad state of affairs – that while Rails and Django might make CRUD easier, that the ORMs weren’t suitable for scaling beyond “toy” sizes, and that more importantly, they didn’t seem to address almost any of the pain points of building and maintaining a modern website.

A couple recent posts, most notably Krow’s Scaling, Systems Required list, but also Tom Kleinpeter’s post asking Where Are the AB Testing Frameworks? reminded me that I had made my own list. I was originally going to start working on these, but since I’ve now been side-tracked by a few projects, I thought I’d put it out there before it gets too completely irrelevant.

I’ve split this into two sections. The first I call “below the line,” which are more system level (some things straddle the line):

  • API Metering
  • Backups & Snapshots
  • Counters
  • Cloud/Cluster Management Tools
    • Instrumentation/Monitoring (Ganglia, Nagios)
    • Failover
    • Node addition/removal and hashing
    • Autoscaling for cloud resources
  • CSRF/XSS Protection
  • Data Retention/Archival
  • Deployment Tools
    • Multiple Devs, Staging, Prod
    • Data model upgrades
    • Rolling deployments
    • Multiple versions (selective beta)
    • Bucket Testing
    • Rollbacks
    • CDN Management
  • Distributed File Storage
  • Distributed Log storage, analysis
  • Graphing
  • HTTP Caching
  • Input/Output Filtering
  • Memory Caching
  • Non-relational Key Stores
  • Rate Limiting
  • Relational Storage
  • Queues
  • Rate Limiting
  • Real-time messaging (XMPP)
  • Search
    • Ranging
    • Geo
  • Sharding
  • Smart Caching
    • dirty-table management

The second section, which I call “above the line” are common application level components that typically depend on one or more of the components above. There are of course a huge list of features for any component, but I’ve highlighted some that either aren’t commonly implemented or are particularly important:

  • AuthX (AuthN + AuthZ)
    • Capabilities
    • Multifactor Auth
    • Rate Limiting
    • Signup
    • OpenID
    • OAuth
    • External import
  • Groups
  • Invites
  • Lists
  • Notifications
    • Spam filtering
    • Multi-protocol routing
    • Fine-grained controls/rules
  • Presence
  • Social Activity Log (Newsfeed)
    • Filtering
  • Social Model
    • Connectivity (uni/bidi)
    • Privacy (private, reciprocal, public)
    • Views
    • Traversal
  • Social Object
    • Privacy, Social Scoping
    • Voting
    • Sharing
    • Publishing
    • Comments
    • Favoriting
    • Social editing
    • Permissions
  • Tagging
    • Combinations
    • Relatedness
  • User
    • Achievements/Awards
    • Activity Log
    • External User ID Mapping
    • Permissions (see AuthX)
    • Deletion/Archival
    • Flagging
    • Direct Messaging
    • User Cards

This list is by no means complete, but maybe a good starting point. I’d be interested to hear what other people have had to build/would most miss if they had to start anew.

(What seems the biggest shame to me is that everyone is currently rebuilding this stuff over and over again and rationalizing it as some sort of secret sauce competitive advantage when it’s really infrastructure – stuff that really should be standardized so you can actually get around to doing the new and interesting stuff.)

Update: For those of you who feel the urge to comment about not needing this functionality: if existing frameworks work for you, that’s great. Also, if you’re not building a site that provides a service to users and have or are planning on being able to grow it, then you’ve likely not faced these pain points. Feel free to move along.

Now, I would like to hear from others working on similar problems, although I understand that most of those people remain under the corporate veil where this sort of information remains “competitive advantage.” Hopefully putting this list out there helps people realize that everyone’s building the same stuff over and over again (to varying levels of quality).