What Will It Take to Form a Cloud Computing Market?

A few weeks ago I joined the Google Groups Cloud Computing group as a charter member of sorts. Today there is an excellent thread going on regarding the various elements needed to make a cloud market work. It started with Reuven Cohen of Enomaly proposing the need for a new marketing term...er, technology concept...he calls a Virtual Private Cloud. The idea here is to create a logical container for a variety of resources located across multiple data centers and technology, making it all appear as a single homogeneous computing environment with security, management (SLAuto?), etc.

However, what has spawned from that original proposal is a wide ranging, but thoughtful discussion of what is needed to allow for an open market for compute resources in the cloud. Participating are several of the vendors supplying various services for Amazon EC2 today (and, I'm sure, a wide range of others in the future), and several end users of primarily "Computational Grid" technologies.

I sent several responses. Here are some highlights:

• Mike Culver pointed out that

  "Condor (http://www.cs.wisc.edu/condor/) is a project that enables this sort of scenario. But in so doing, things go full sircle [sic], and suddenly the paradigm is the old days of mainframe computing where the notion of "job" is separate from the underlying computing resource."

  I replied

  "I hate the notion of every software executable being a "job". [With most] high availability applications running additional instances [of "permanent" processes] in excess capacity at another site is a distinctly possible scenario.
  
  I prefer the term "software payload" to describe what gets moved from cloud provider to cloud provider, at least at the HaaS level."

  The idea of a "job" is just not applicable to many user-facing applications.


• Larry Ludwig of hosting provider Empowering Media notes

  "Application scaling IMHO will always
  involve a mixture of automated systems and programing changes to the
  application. I don't think this aspect of cloud computing can ever be
  completely automated.
  
  The typical "throwing hardware at it" works up to a point and in cloud
  computing will be no different since the a cloud system is still based upon
  the Von Neumann architecture. There is a point where it becomes more than a
  sysadmin/scaling challenge. Programming changes will need to made to the
  specific application. What scales to X, won't scale to Y because of a
  different bottleneck."

  And I agree, adding

  "I see two architectural problems to be managed when building an application for the cloud:
  
  - Scalability - which you cover below
  
  - Fluidity - which is the ability to move an application, *application
  tier* or service between cloud infrastructures without rewriting or
  reconfiguring the software payload"

  I blog extensively about software fluidity in these posts.


• Geoffrey Fox notes that much work has been done to analyze and design Computational Grid economies. I assumed that Geoffrey was taking about grid computing models based on splitting up "jobs", so I noted:

  "The problem is partially that Computational Grid computing is a subclass of the Cloud Computing metrics and standard problem. See earlier notes about long running processes versus "jobs"."

  In other words, I'm not sure how far people have gone to analyze enterprise computing on a grid versus just HPC, etc.

There is so much more to read on this thread and others in this rapidly growing group; not the least of which will be the responses to my thoughts here. If you haven't joined yet, and you are interested in cloud computing strategy and tactics, I recommend that you get involved.