random($foo)

Infrastructure for Modern Web Sites #

random($foo) is the personal site of Leonard Lin, where I collect shiny things and publish original writing and code. more »

One of the things that I did when I wrapping up at Yahoo! was to begin to take a look at the current state of web frameworks. I ended up picking Django, but I have to say, I was disappointed with the state of what’s out there. Friends will have heard me bemoaning this sad state of affairs – that while Rails and Django might make CRUD easier, that the ORMs weren’t suitable for scaling beyond “toy” sizes, and that more importantly, they didn’t seem to address almost any of the pain points of building and maintaining a modern website.

A couple recent posts, most notably Krow’s Scaling, Systems Required list, but also Tom Kleinpeter’s post asking Where Are the AB Testing Frameworks? reminded me that I had made my own list. I was originally going to start working on these, but since I’ve now been side-tracked by a few projects, I thought I’d put it out there before it gets too completely irrelevant.

I’ve split this into two sections. The first I call “below the line,” which are more system level (some things straddle the line):

  • API Metering
  • Backups & Snapshots
  • Counters
  • Cloud/Cluster Management Tools
    • Instrumentation/Monitoring (Ganglia, Nagios)
    • Failover
    • Node addition/removal and hashing
    • Autoscaling for cloud resources
  • CSRF/XSS Protection
  • Data Retention/Archival
  • Deployment Tools
    • Multiple Devs, Staging, Prod
    • Data model upgrades
    • Rolling deployments
    • Multiple versions (selective beta)
    • Bucket Testing
    • Rollbacks
    • CDN Management
  • Distributed File Storage
  • Distributed Log storage, analysis
  • Graphing
  • HTTP Caching
  • Input/Output Filtering
  • Memory Caching
  • Non-relational Key Stores
  • Rate Limiting
  • Relational Storage
  • Queues
  • Rate Limiting
  • Real-time messaging (XMPP)
  • Search
    • Ranging
    • Geo
  • Sharding
  • Smart Caching
    • dirty-table management

The second section, which I call “above the line” are common application level components that typically depend on one or more of the components above. There are of course a huge list of features for any component, but I’ve highlighted some that either aren’t commonly implemented or are particularly important:

  • AuthX (AuthN + AuthZ)
    • Capabilities
    • Multifactor Auth
    • Rate Limiting
    • Signup
    • OpenID
    • OAuth
    • External import
  • Groups
  • Invites
  • Lists
  • Notifications
    • Spam filtering
    • Multi-protocol routing
    • Fine-grained controls/rules
  • Presence
  • Social Activity Log (Newsfeed)
    • Filtering
  • Social Model
    • Connectivity (uni/bidi)
    • Privacy (private, reciprocal, public)
    • Views
    • Traversal
  • Social Object
    • Privacy, Social Scoping
    • Voting
    • Sharing
    • Publishing
    • Comments
    • Favoriting
    • Social editing
    • Permissions
  • Tagging
    • Combinations
    • Relatedness
  • User
    • Achievements/Awards
    • Activity Log
    • External User ID Mapping
    • Permissions (see AuthX)
    • Deletion/Archival
    • Flagging
    • Direct Messaging
    • User Cards

This list is by no means complete, but maybe a good starting point. I’d be interested to hear what other people have had to build/would most miss if they had to start anew.

(What seems the biggest shame to me is that everyone is currently rebuilding this stuff over and over again and rationalizing it as some sort of secret sauce competitive advantage when it’s really infrastructure – stuff that really should be standardized so you can actually get around to doing the new and interesting stuff.)

Update: For those of you who feel the urge to comment about not needing this functionality: if existing frameworks work for you, that’s great. Also, if you’re not building a site that provides a service to users and have or are planning on being able to grow it, then you’ve likely not faced these pain points. Feel free to move along.

Now, I would like to hear from others working on similar problems, although I understand that most of those people remain under the corporate veil where this sort of information remains “competitive advantage.” Hopefully putting this list out there helps people realize that everyone’s building the same stuff over and over again (to varying levels of quality).

Tags: , ,

This entry was posted on Wednesday, January 28th, 2009 at 8:14 pm and is filed under MakeItBetter, Tech, Web. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

  • How about making Facebook, OpenSocial and/or iPhone apps out of the box? For me a web _site_ (meaning standalone and mostly accessed while sitting in front of a computer) seems like an iffy proposition going forward.
  • I think Michal Migurski was right on with the point that we're going to be building modules and sub-modules on different domains. How much looser can your coupling get than that. Furthermore, we're going to see the remote services (S3, Google App Engine, ...) continue to differentiate themselves, making a one stop shop hard to come by. But, the way things are going, why would you want a single centralized service anyway. Something that has not been mentioned yet, is the huge push to "upgrade" the clients, both computer, and mobile. (Gears, Silverlight, Air, HTML5) So I see two problems that need to be addressed here:

    1. Monolithic frameworks solved the last problem... we need something to solve the *next* problem. We need a bunch of people that can put the cognitive hurdle that Rails and Django overcame behind us. Learn from them, and move on. "If Henry Ford had built what everyone wanted, we'd all be riding better horses."

    2. With the emergence of "fat" clients and data services coming from every direction, we're going to need a GNU like approach (this has been mentioned several times in the comments). A lot of small, loosely joined parts, held together with a brilliantly simplistic bit of glue.

    Let's start building some more use case lists, and come up with something we can't break. I don't buy into the idea that a framework is relegated to being jack of all trades and master of none. I view Linux, and even the web itself, as "frameworks".
  • Dan Creswell
    A pretty good list - how much of this stuff is anyone prepared to pay for and how much are they willing to pay? Think this is starting to get to the point:

    "Let's not forget that the existing frameworks were created to solve what was at the time, to most people, "complex stuff". The fact that we are calling those problems "simple scenarios" now is surely a testament to how these tools have helped people. Why not continue making the complex simple?"

    These bits of infrastructure aren't so easy to build as the offerings of yesteryear which leads me to wonder about the economics. Perhaps the only way to develop these things in reasonable time is via paid development, perhaps we're beyond what can be done at home by an individual on weekends and spare evenings. Perhaps we're at a stage where to develop a good chunk of this infrastructure requires a startup and finance which is a serious undertaking. Perhaps if developers want this stuff they'll have to persuade their company to pay for it.

    I suspect also that the mindset of the average developer is limiting - most prefer to develop with what they know leading to the usual strongly consistent, failure intolerant, badly architected homebrew solutions. Should they develop frameworks or infrastructure they develop it with that mindset which means that much of what you list above is left un-done or done badly.

    Just beginning to think we've got a few substantial barriers to progress beyond the mere technical.....
  • lhl
    Dan, that's a pretty interesting point that you bring up, and part of why I think the current frameworks have been so flat-footed about it - it's been reflected somewhat in these comments - those that can relate these problems are typically busy working in industry, and those that don't are pretty oblivious about it.

    At the same time, for most of the elements that I've listed above, there large chunks of the most difficult pieces that already exist as open source components (which continue to get open sourced for all the reasons that things get open sourced), so the complexity difference isn't *that* big a leap. Many of these elements have only really matured to the point of usability in the past year or so, and that in combination w/ the current wave of the technological cycle I think actually makes it a very good time to think about this.

    I was going to make this as a topic of a separate post, but I guess I can just post it here. What we have now is a set of patterns that is now fairly well established, but is still at the "artisanal" level - that is, they can't really be easily put together without a team going through a large number of components - at the same time, there are now good building blocks for building these sub-systems, and I think that with the turnover of people that have built these systems for proprietary systems are going to realize that they're going to need to build it all over again. A few of them I suspect will get together and build this as open source so at least they won't have to rebuild it next time, no matter where they end up working at.

    That, I think is the natural cycle of how these open projects are driven, and a big part of the things that came out of the last wave...

    Anyway, my hope is that this discussion might serve as a little push in that direction because the biggest barrier I see isn't technological but really the philosophical underpinnings/mindset of what a framework should be and should look like.

    I.e., even in the comments where say someone might agree about supporting some of these new components, there's the simultaneous talk of additional functionality making things "slow" or "heavy" - which I think is a reflection of the weaknesses of the monolithic, tightly coupled design philosophy that Rails (and, despite protestations otherwise, Django - it's not *really* loose coupling if everything stops working when you switch something out) and other frameworks have tended towards. In any case, I don't think the level of modularity and decomposition/separations need to be rethought especially as more and more pieces of functionality are independent/virtual subsystems.
  • Yes!!!!

    I've had a few jobs in the past couple of years and every time I ended up doing much of your list over for a new framework. It's annoying and as an industry a HUGE sinkhole of time and money.

    I especially liked the comment about the image manipulation as an external service. If many of your components could be developed as independent, well designed services I'd be likely to use them. However if Rails or Django were to implement them all internally I would never use the framework, preferring a lighter one because I too get very wary of kitchen sink frameworks. In general, the bigger they get the slower and harder to modify when things break down they become.

    To add a little weight to the services side just take a look at what's happening on blogs with ReCaptcha or even here with Disqus. People are tired of writing and rewriting comment systems and captchas so they just integrate a 3rd party's system. I have a feeling much more of this will be going on in the future.
  • Really? We're friendly! We need an expert to talk with. Scott and I can't just make this stuff up--although some people wonder. :-)
  • Would you like to come on Technometria (IT Conversations) podcast and discuss this list next week? Contact me, if so: http://xri.net/=windley
  • lhl
    Phil, I'm not much of a podcast guy, but feel free to use this list as starting point for a conversation if you like. It's been interesting to see the variety of responses so far...
  • I agree that everyone is building this over and over again. To date, most web frameworks (and Drupal is the one that I've been most involved with) have had very little involvement from sysadmins. I find it is mainly the "sysadmin tier" that builds this stuff over and over again, usually out of shell scripts and duct tape. And they come up with some fantastic systems, they're just not easy to replicate because they, themselves, don't sit inside any sort of framework that makes it easy to re-use or extend.

    So, the forward facing web frameworks could learn a lot from making some back end pieces more pluggable.
  • Couldn't agree more! That said I love Django (specially coming from a Java background). It does offer a good balance between whats part of Django and whats not. 2 examples: caching backends and storage backends (100 lines will get all your data neatly into S3)

    I do agree many of these things could be added to django.contrib specially database backups (to S3), distrubuted file storage etc. Then again frameworks have a problem, something for everyone or everything for a few!
  • Kai, the chubby lock server is the missing piece of the triforce over at AWS ... if they had such a thing, S3 and SQS would instantly become 10x as useful, as "serverless" services.

    I'm reading some of the comments further up, and the image thing piqued my interest. This seems like a situation best handled by a more unixy, many-small-tools approach. For example, on Oakland Crimespotting (http://oakland.crimespotting.org) we have static maps showing locations of crime reports, police beat histories, etc. These all come from a tiny web server (a modified http://modestmaps.com/examples-python-ws/) whose sole job is to cough up pictures when asked. It doesn't know about the database, and doesn't need to. The PHP application doesn't know how to make images, and doesn't need to. It's an excellent separation of concerns, and it's future-proofed to boot: should the traffic ever become excessive, we could move that component off to a different server someplace else, with no complex adjustments except a new URL. Tiny web servers everywhere, is where I think this stuff needs to go. I'm not altogether happy with the kitchen-sink model promised by Django, Rails, etc. - they feel a bit like buying bicycles with the training wheels welded-on. Completely fine for a wide range of needs, but not built with growth in mind.
  • lhl
    Yeah, I'll be doing a followup post soon - I think one of the things throwing people off is that today, when someone says "web framework", it's viewed as a monolithic library that gets loaded and responds synchronously when they should be thinking about an interface to a set of components / virtualized subsystems that run as separate services.

    The welded-on training wheels is a great metaphor. To extend that metaphor, you typically need to take those training wheels off when you can least afford to stop. (and find a blowtorch?)
  • AaronHeld
    There is an interesting choice here between build and buy. Many of the 'above the line' services exist as third party / ajax type plug ins that reduce your dependency on building infrastructure. (I notice I'm entering a comment into such a system provided by Disqus)

    I recommend reading the 'innovators dilemma' around this topic. The market pressure drives me to use existing vendor plug-ins, but at the end of the day if you have no infrastructure then you have lowered the barrier for people to compete with you.

    After all, if you build your app as a Facebook app you will get much of this for 'free'.
  • lhl
    Using a framework is different (at least in degree) from ASPing because in the latter case you need to consider who you're beholden to and typically, the impact on your ability to modify is much greater.

    I'm quite familiar with the innovator's dilemma, but I'm not seeing how it applies - there's no technological disruption going on per se, this is, as you mention, in business terms, just an old fashioned barrier to entry.

    Anyway, at the end of the day, as a business, sure you're need to be able to provide a "value add." The question I think that's salient, is that given limited technical time/resources, whether you're better off recreating things that have been done repeatedly, or if your competitive advantage should come from creating something new. (That question is a bit misleading of course, since my point is there isn't the option of *not* recreating the things that have been done repeatedly. :)
  • Great post, I agree, though for the most part, I think ORMs to a large degree are tough to make correctly (even in Django) and leave a lot of things highly unoptimized. I am usually paranoid about the ORM sucking and leave only simple queries to the ORM and complex ones to me writing them out.

    Also, regarding A/B testing metrics, I'd hope I am contributing to people not having to reinvent the wheel these days via a startup of mine: http://mixpanel.com . I wrote a post regarding A/B testing: http://blog.mixpanel.com/2009/01/ab-testing-your-website-with-mixpanel/
  • Kai
    A lockserver similar to Google's Chubby would probably be pretty useful. It seems to occupy a niche not provided by any of the other stores and can act as a backbone to coordinate the other services.
  • Kai, you might want to take a look at ZooKeeper over at Apache - http://hadoop.apache.org/zookeeper/
    "ZooKeeper is a high-performance coordination service for distributed applications."
  • jacobian
    Thanks for the list, Leonard -- it'll make a good reference the next time I need to explain why writing a quick application is really just the *start* of a web developer's job these days. Like Ned I've cobbled together most of this stuff myself. Indeed; I've hacked together some of those "below the line" components at least three times.

    For me, though, the really interesting part is that it seems you're arguing that these parts ought to be part of a modern web framework. And you might think I'd agree given the amount of time I spend on this stuff... but I'm not sure I *do* agree.

    See, I tend to think of this stuff as "the rest of the stack" for a good reason: it all tends to be *very* domain-specific. "Sharding" is an easy shorthand for a common feature, but I suspect if you inventoried a bunch of large web apps you'd find each has a substantially different interpretation of what a "shard" actually is. Case in point: I've designed or implemented four different somethings I'd call "sharding" on a resume, but really the similarity stops at the umbrella term. I think it'd take more time to route around an almost-certainly-useless set of defaults than just to start from scratch each time.

    I think the problem is that when you start to write down the commonalities in these "below the line" bits of different apps you come up with a *really* small overlap. For me this is most striking when it comes to deployment: having tried a bunch of different deployment "frameworks" I'm starting to realize that the only common feature set I need is ssh and a for loop. Anything else I inevitably just have to route around or rewrite eventually anyway.

    It seems trying to implement the "below the line" features in a common way would just be an exercise in frustration.

    I think what might be needed here is some sort of "framework distribution" -- Django/Rails/whatever plus all the "other stuff" you'd use in the Real World. Like the "GNU" in "GNU/Linux". After all, all these "below the line" tools already exist in various and sundry forms, it's just that (in grand open source tradition) they're not integrated. A determined party could pick a best-of-breed from each category, write a bunch of wrappers, configuration, "glue", and GUIs and ship a distribution (or launch it all as a hosted platform). I think it's only a matter of time, really.
  • lhl
    Jacob, the GNU/Linux metaphor I think is pretty apt. There's a pretty diverse collection of systems/tools that is more along the lines of a Linux distro than the current way that web frameworks are packaged.

    I agree that there are large pieces of functionality that exists already for the below the line tools - NG web frameworks should aim to allow those to be coupled easily, and probably to provide some additional management functionality. In some cases, there probably exists an opportunity to build some useful functionality to fill the gaps. Offline/asynchronous task processing for example is one of those things. There are quite a number of queuing back-ends available, but for say an overall system of feeding and processing of tasks, I myself have built two (one for Upcoming and one for MyBO), and I can easily rattle off half a dozen friends I know who have built their own. Are they that different? I'm not so sure about that.

    So a couple things about the current state of web frameworks. One which we've been talking about is scope. If a framework doesn't address this sort of stuff, what is the value? Moving dispatching from mod_rewrite to the application layer? Making CRUD forms a little easier? I think that we agree that for most of the projects that really need a framework (I don't know a better way to put it, I know the term "modern" rubbed some people the wrong way - maybe Cal's "serious" really is the best one), it's those *other* bits that require most of the work.

    A second thing, is that a big frustration is that the way that the current web frameworks are packaged, they're actually developer hostile when it comes to putting together those other bits of for growth. For example, with Django, while you get these admin tools and other niceties out of the box, because they're all based on the current ORM, you're completely at its mercy for support for sharding/federating. Last time I looked (a couple months ago, I've been a bit distracted), the ORM didn't support slaves even. This is just one example, but the frameworks I've used are littered w/ stuff like this - things that are optimized for getting up and running, but end up causing much more pain than when you need to scale up. This of course usually happens at the most inopportune time, since you experience this crunch right usually as you hit the hockey stick. Sure sharding may be something that's somewhat domain specific, but having built in support for that in your data layer? Seems like a good idea.

    (Also, as an aside, I'm particularly biased against ORMs anyway since it's easy to squeeze 100-1000x performance gains from bad SQL, but maybe that's neither here nor there.)
  • jacobian
    Yeah, the more I mull it over the more a "framework distribution" seems like a good idea. Maybe I'll even make some time to work on one.

    You're right on, though, that "NG web frameworks should aim to allow those to be coupled easily, and probably to provide some additional management functionality." Take Django's admin being hostile to -- er, more like almost completely incompatible with -- third-party ORMs: we're in 100% agreement that this is more like a "bug" than a "feature." It's actually one of my top priorities, along with about three hundred other things. (I think I need a better way of categorizing my todo list.)

    A framework with hooks for "the rest of the stack" is a better framework. It won't happen overnight, but my hope is that over the next few releases we'll be able to shape Django along these lines. Sharding/federating/slaves is a pretty good example, actually: Django 1.0 has all the low-level plumbing needed to support multiple databases. All the hooks are there, so what's needed is a bit of a higher-level API for convenience and some documentation explaining how to handle common patterns. We have to do the plumbing first, though; that way the early adoptors can help us figure out what style of porcelain to build.

    I'm hoping we'll be able to follow this pattern in other areas -- introduce the low-level hooks, let folks fool with 'em, and then make the common patterns easy and documented. For example, my current bugaboo is monitoring: Django's pretty opaque about what's going on internally. I'd like to make that data available to external monitoring tools (Munin, Cacti, Zenoss, ...) but what I really DON'T want is to build a full-stack monitoring tool into Django itself.
  • This is a great list. 80% of it describes exactly the sorts of things we've cobbled together at tabblo.com to build up from what Django produces. Personally, I'm less interested in frameworks providing the user-visible stuff (groups, comments, tagging), because the details of how I want to provide it to my users will always differ in the details from the off-the-shelf version. But the infrastructure stuff (counters, bucket testing, log analysis, deploy, rate limiting, csrf xss, etc) is invisible to users, and is often tricky to get to production quality. This is a fabulous manifesto for where the frameworks should be headed.
  • Max
    Hi Leonard,

    after reading your post I was astonished that you are talking about web frameworks regarding to the features you are asking for. Wouldn't the term "CMS" fit better here?

    Max
  • lhl
    Max, I don't think so. The above functionality listed is pretty orthogonal to any "content" to be managed. This list purposely abstracts out any domain-specific functionality. The above the line functionality specifies components that any user-oriented service should provide.

    That being said, I think that what I'm specifying would require a different approach from the current generation of frameworks, which are still pretty monolithic. Most of the challenging parts of creating modern (social, service-oriented) web sites are in putting together the heterogeneous sub-systems mentioned in the first list.
  • Gareth McCumskey
    It sounds like the description of components is more in the lines of CRM's packages rather than development frameworks. Ruby and symfony (a PHP framework) abstract developers from the how of development, such as database interactions, ajax libraries etc. and rather lets a developer focus on getting the design out as easily, quickly and accurately as possible. Building social networking tools into a framework is like developing another CRM packages with modules you can use rather than a light, lean and fast development tool to make your development methodologies easier to accomplish and your designs at reachable.
  • lhl
    I'm not really seeing your distinction. Rails, Django, et al already have admin tools, auth etc. that fall within what you're calling packages (the "above the line" components I mention) and as for the below the line list, that certainly should be within the purview of any framework that'd make my life easier. I don't quite see the point of using a framework over my own codebase otherwise.

    To pick on Django for instance, a big selling point of what it offers is it's data modeling/ORM and admin interface. All this breaks of course if you, oh, I don't know, want to runs slaves or shard. Or if you say, attach it to big table. Or if you start upgrading or try rolling back your data model. You get the picture.

    I think it's worth pointing out that one thing that is different when talking about the various infrastructure components is that you're talking about many more components. Instead of a DB, application server and web server stack, you're adding a variety of additional systems, several of which require asynchronous operation. For all the components that I listed above, there are open-source components (usually at least a couple choices) that can be used, it's just a lot of work to bring it all together. A good next-generation framework IMO would provide loose coupling that would make this easier (or, say in the case of management tools, provide additional functionality to the base services).

    This sort of abstraction layer is exactly what frameworks are supposed to be.
  • Wow late response but I thought I would extend more.

    Let me use the symfony framework as my example. symfony pretty much provides all of the below the line features that you listed, which I think any good framework should do. WHat symfony does do differently is it allows everything else to be selected and installed as a set of plugins that the core symfony development team create or even others of the community can create. Things such as CRM's, social applications, etc. Personally I feel a frameworks core focus needs to be on the below the line features you mentioned top provide developers with a great base at which to begin development and on top of which they can incorporate their own and others work (as plugins in symfony's case) knowing that all that other stuff that is actually so important is already taken care of.

    The problem with incorporating all the those above the line features as part of a framework is that there are so many instances and so many variations and unique differences in the way that various developers,companies and groups of developers would want to implement something like, for example, Notifications that it cannot be a "one size fits all" effort. Attack protection is good for everyone, a Social Model might only be good for a few and then you are limiting your framework to a select few.
  • coulix
    Come on what would be left for us to ? wrapping ?
  • lhl
    Is that a serious question? There's an infinite amount of stuff to be done/made better. Just imagine all the time that you would have spent reimplementing infrastructure and apply it to either extending it or working on your domain-specific functionality.
  • An excellent survey.

    Assuming that you haven't listed basic modules in most stacks, I'd add a few specific modules below the waterline:
    Ajax
    Comet
    Image conversion
    Audio and video conversion and streaming
    Spatial simulation (for 2D and 3D spaces)

    Above the waterline I'd add:
    Asynchronous HTTP messaging with channels, filters, ... (until XMPP is in browsers)
    Text chat
    Audio groups
    Image editing widgets (to crop avatars, tag images, ...)
    Video rendering widget
    2D and 3D space rendering widget
  • lhl
    Trevor, image processing, storage and serving is a pretty good one. Definitely something everyone has to cobble together over and over (think custom avatars at least).
  • Tagging is not complex. Non-relational key stores are not complex. Alerting is not complex. Most consumer web services needs these. Every builder of a consumer web service wastes time building these. They do belong in a framework (or more correctly as modules in a framework).
  • igorgue
    "Framework Design Guidelines" (http://blogs.msdn.com/brada/archive/category/2656.aspx) states: "Simple things should be easy, complex things should be possible" (the book uses "scenarios"), these, IMHO are complex things, they are possible some easy some not, but they can be done.

    In the end how many websites need all these ponies? 1, 2? maybe in your website you'll need 3 of all the ponies, I rather make the all the simple scenarios easier than adding more complex stuff to the framework.
  • Let's not forget that the existing frameworks were created to solve what was at the time, to most people, "complex stuff". The fact that we are calling those problems "simple scenarios" now is surely a testament to how these tools have helped people. Why not continue making the complex simple?
  • lhl
    The point is, without the below the line functionality, how does a framework make your life easier? With an URL parser?

    Maybe you can assume that you won't have to worry about scaling your site in the future, but w/o which, I don't see how a framework provides much advantage to a site w/ any sort of ambitions (at least not w/o major rewrites at the most inopportune times). The deployment scenarios outlined above are absolutely fundamental - upgrades and rollbacks? Come on, I can't think of any website that *wouldn't* need these.

    To clarify, I'm not saying that every site needs all of the outlined functionality, and I can see how these frameworks can help people that are just getting started or for a certain scope (although a large subset of those could probably as easily fulfill there needs w/ running on WordPress, Drupal or Joomla), when I have been looking at developing and growing a new project, these are some of the common requirements (and pain points) I see over and over again.

    And, as the frameworks out there don't address them, they don't really make life easier for me than the PHP code I had (which *does* have a fair amount of this and which otherwise gets out of my way). This is frustrating for me. In my circle, I see it as a big gap because everyone who's starting over is also forced to rebuild much of the same things.

    If you're well served by what your framework provides, good for you, but when you end up implementing the stuff I've listed (particularly the below the line components), just remember that it's been done before, probably with a lot more man hours put in. Many times. Over and over.
  • I couldn't agree with you more, Leonard.

    Existing frameworks typically take a lowest common denominator approach in order to appeal to the broadest market. That market is typically time and funding constrained. Successful applications can quickly outgrow these frameworks.

    Perhaps broad-scale adoption of cloud infrastructure and cloud services will make more of what you describe readily available to those that need it.
  • Most of the 'above the line' components you request are available as part of Pinax http://pinaxproject.com/ today
  • lhl
    Yeah, Pinax is interesting (on the PHP side Elgg and Drupal also implement a lot of functionality), thanks for bringing it up.

    I think the weaknesses for all those projects are that while they'll get you up and running, scaling up or running in a production environment means you still have to tackle all the below-the-line-stuff yourself, and/or probably rewrite most of the components. Given the appropriate infrastructure, it's not *that* much harder to write the equivalent component that would scale, but for whatever (many) reasons, it seems that that just doesn't really make it's way back to the other side.
  • jtauber
    The above-the-line/below-the-line distinction is one of the reasons that I think a Pinax+Django division of labour is better than a single framework that tries to do both. Your post is possibly the best example I've seen of the distinction I'm shooting for between Pinax and Django.

    And your final paragraph is priceless. Mind if I quote it in Pinax talks? Actually can I quote that on the Pinax website?
  • lhl
    Hey James, go for it.
blog comments powered by Disqus