The Cloud Marketplace - All Resources Welcome

There is a singularity in the future of cloud which I believe has guided its evolution since server virtualization became a topic in the early 2000's. However I only realized today that not everyone sees this inevitability. Cloud is about agility, efficiency, and elasticity where the low hanging fruit, cost savings, has brought a by-product of the transformation to the forefront as its value proposition.  This in turn has focused our attention on the infrastructure side of cloud largely ignoring the original value proposition; the global collaborative applications composable from services and available anywhere, anytime, on any device.  So today we see cloud as an enabler of new models for infrastructure consumption matching hardware needs to cost: on premise, off premise, private, public, hybrid, IaaS and even PaaS and SaaS.  The product mix is typically viewed as mutually exclusive however in the optimal cloud it's not about one or the other, it's about having all of them.  Only by maximizing our consumption option can we tailor available resources to our needs.  I believe this road leads to one end state: an automated, autonomous cloud resource marketplace.

What on earth is that?

I admit I jumped to the punch line so let me make my case in a more linear fashion starting with an easier topic: the automobile. We use autos everyday to achieve many of our objectives: getting to work, picking up children, traveling to a client. At home we may own a car but when away from home we often have the same needs. And there are many who don't even own a car but need one from time to time. It's no surprise businesses have grown to meet these needs whether it's use a car for a few minutes (taxi), a few hours (asset shares like ZipCar), a few days (car rental), a few years (lease) or forever (purchase). The need for the resource drives the consumption model at any given time. That need is composed of a variety of factors that are contextually dependent such as where, when, for what purpose, for how long, with whom, etc. We balance these needs through prioritization and look to the market to provide the solution: a bullet-proof limo for the President, a rented mini-van for the family on vacation, a leased truck to deliver furniture, and a taxi cab to run downtown for a meeting.

We need the same market options in technology but so far there have been serious limitations. What companies need is the ability to benefit from technology without having to buy and become experts in it.  However due to the limitations of the market, most companies have chosen to own and operate their technology.  As a result the value of each dollar invested in IT is diluted by the cost of moving up the knowledge curve which does not lead directly to improved revenue or reduced costs.  The first big step out of own and operate was mainframe time sharing which quickly led to outsourcing, a model of own and let someone else operate. It's such a small step there really isn't an equivalent in the more general market which probably explains all the pains behind outsourcing.  Furthering the adoption of alternative models we've had some success with co-location (lease model), Application Service Providers and now Software as a Service (rental model), a we're nibbling at the shared asset model with public IaaS.  However large scale adoption of a shared asset model is years away and there is no existing path to the taxi cab model due to issues of security, regulatory compliance, trust, and a healthy dose of fear, uncertainty and doubt (FUD).

Today we are at nearing the peak of the hill and it's time to start planning for what happens when we crest.  Our journey into cloud isn't about adopting new models and leaving old ones behind.  Cloud is about doing it all; the new and the old.  Top to bottom.  It's about the right resources at the right time in the right place.  I believe a real cloud can deliver any kind of resource required from a mainframe processor to heat monitor's data. However getting to this ubiquitous cloud requires a fundamental shift.

As an infrastructure technology today, the vision for cloud is, at best, cloudy.  The entire purpose of what we call Information Technology is to execute applications.  To an application everything is a resource.  Therefore it follows that from an application point of view, a cloud is all about the marshaling of resources (compute, memory, data, algorithm, bandwidth, security, etc.).  There is no one size fits all answer, and as our applications become more user-centric by nature they will exhibit unique tendencies at the user level requiring micro-levels of optimization.  Hence the need for all resource types and consumption models which enables cloud solutions to leverage the best available combination of resources at any given moment based on what's available. Continuous optimization becomes a scanning routine constantly looking for better resources or a change in the parameters.  Inherent in this approach are all the benefits of cloud we know and love: scale-up/scale-down, pay as you go, no long term contract, etc.  For lack of a better term I call this the Cloud Marketplace (NOTE: I am advocating the marketplace as an architectural pattern which would, by nature, enable the cascading of marketplaces to maximize the resource and consumption model pools.

Using the car analogy this approach would allow us to use the best vehicle possible for each task (go to work, visit a client, pick up the new widescreen TV, go out to dinner) rather than compromise by trying to fit all the uses to a single model.  There is no convenient, economic model today that can get us a hybrid to get to work, a town car for the client visit, a delivery truck for picking up the TV, and a sports car for a night out. Luckily in the virtual world the only barriers are those we create.

A cloud marketplace would be governed by a set of business attributes (time to market, capex sensitivity, flexibility, self serve, service quality, criticality, compliance, IP protection, and continuity) and  technical attributes (cost, location, availability, execution speed, latency & bandwidth, integration, reliability, security, scalability).  Each application would include in its profile a predefined prioritized list of attributes to be used for by the marketplace.  To a well designed cloud app there is no dependency on the infrastructure so the marketplace would use the attributes to marshal the necessary resources via the orchestration and provisioning systems to get the compute, memory, bandwidth, and other resources required. When the parameters change (time to market drops from high to medium) the system would look to modify the resource pool (move from $0.08/CPU/hr to $0.05/CPU/hr compute) to comply.  The power of a marketplace model is in its inherent organization and optimization.  Resources can be ranked by value based on real usage patterns providing clarity in capacity decisions.

No Fortune 500 company is going to rush out and close down data centers throwing away billions of dollars in investment to adopt this pattern.  And they shouldn't.  The concept naturally lends itself to a stepwise adoption.  Once we have applications unbounded to resources (virtualized in some way) and can operate across more than one set of resources we have a need to manage and optimize.  The marketplace can be built over time with self-identifying resources enabling plug-and-play consumption models.  The more consumption models it supports the more resources made available for use.  Once we have a marketplace we can expand our thinking on what constitutes a resource.  Imagine a future where data is collected and stored at the endpoints, where compute power and connectivity are ubiquitous. Taking advantage of these resources in a marketplace is simple - just define their parameters and make them available.  Without the marketplace concept the risk is likely too high to consider.

To get to this point we have several challenges.  First we need to eliminate dependencies.  Dependencies between resources.  Dependencies between applications and resources.  The fewer dependencies the better the marketplace operates.  Second, we need better security tools able to define, execute, and audit policies throughout the execution cycle of an application.  Third we need application management tools that are non-intrusive to the application which in turn requires a new standard for communicating application state.  Fourth we need to address data management and security in a federated model.

To make such a model work requires a level of insight and automation well beyond where we are today. No such marketplace exists today and I'm sure it's not on any product roadmap. However I'm convinced it's the vision we need to work toward. Cloud is a global collaboration platform, nothing more but also nothing less. The power of cloud is delivered by the applications built to take advantage of the platform: Social Business, Real Time Analytics (aka Big Data), Mobility, and Collaborative Communications. The reason to adopt cloud is to bring our connected world to the next level: the collaborative world. Doing so requires a platform that delivers continuous availability, unlimited scalability, and optimal efficiency at all times. To do this cloud needs options and the ability to match need to resource. Anything less isn't a cloud; it's just another silo.