SOA and compute clouds point to rethinking data entirely: roles and permissions, not rows and tables

There's nothing like a conversation to bring out insights and new ideas. Tony Baer and I were chatting on this very pearl of productivity last week, that an open roundtable analyst call always allowed us to move the needle forward in terms of thought leadership in ways that solo writing and one-think cannot.

And then there's Twitter, which is sort of a blend of a roundtable chat, solo writing, one-on-one and blogging -- all to and with a refined, social graph-defined audience. And it was in this act of Twitter-thinking this morning that I slid into a new realization, new for me anyway. It has to do with data, and the need for multi-permeable access to data, across organizational boundaries, stored in many places, that both protects valued and proprietary data while breaking processes out of the corporate IT straight jacket.

I'm seeking better understanding of how cloud (public and private)-base webby apps can and should be a big part of SOA (especially greenfield services), and may even become the driver for rapid use and adoption of SOA. I'm also fascinated by PaaS, SaaS, IaaS, AWS, GAE, and the services ecology development from both small and large providers. And I know that social networking and new media will play into big business in a big way.

These issues have huge impact on many vendors, from Salesforce.com, Amazon, Microsoft, IBM, Google, Yahoo!, Oracle, SAP, and HP, on down to a burgeoning class of startup-class of cloud capabilities supporters. Notice how I lumped together older-style enterprise IT vendors and "consumery" services providers?

The hang-up on understanding how these two worlds -- public clouds and enterprise IT (soon private clouds) -- come together hinges back to who/what handles, owns, manages and offers secure access to (or not) the data.

It's clear that applications and logic can become services, atomized or aggregated. User interfaces can be delivered as a service, as parts of apps or as entire desktops. Same for mobile access.

Of the various levels of abstraction of what goes into IT-based activities, almost all can be deconstructed and more productively delivered as thick, thin, or mixed (software plus services) webby apps. Agile business processes are the new coin of the realm. The days of standalone, PC-based apps are over. Off the wire services will make up more and more of what we will soon colloquially refer to as productivity applications.

But then there's the data, still strapped into a definition that associates it to applications, even as applications as we know them are evolving dramatically. After talking with Dave Linthicum, now CEO of StikeIron, on his vision ... and Kirill Sheynkman, president and CEO of Elastra (as well as many others in the Enterprise 2.0 space) ... I'm now convinced it's time for some radical re-thinking on data, and what it is that it should actually be related to and associated with.

Perhaps it's time to fully divorce data from applications, and wed it all instead to people and groups, guided by roles and permissions, and therefore no longer co-located with applications or even enterprises. House it where it can be used easiest, and stored and protected cheapest. It's heresy today perhaps, but more of the data that matters will be in the cloud anyway.

We always think of data as tied to an application, or in a managed store (often times distributed) that applications read-write to. The store can be analyzed, protected, backed-up, consolidated and cleaned up, messed up again, poked and prodded. We still think of data belonging to some store, and by association, to a department or corporation or IT sys admin.

Data is controlled and managed as centrally as possible, except that it's always scattered and inconsistent. A battle rages in every enterprise as it tries to manage and control its data. It's a losing battle that is costing more and more of the IT spend pie each quarter. And there are many good reasons for this battle -- except that it's holding us all back.

It's now clear that the current mentality of data and its place is holding us back in unacceptably unproductive ways. In this day and age, you will never control your data at the margins. Those margins used to be PCs and departmental servers. Now they are becoming clouds, social networks, free web email apps, Twitter. The data that impacts and drives your company is a complex system not unlike the weather, or quantum-level particles. Try and grab and control that rainbow, if you will. Probability rules, not exactitude.

CXOs can define the happenings of their enterprise by the audits, create the ledgers, and fill up the financials data warehouses. But so much more is going on beyond the glass rooms. The best that CXOs can hope for is to approximate the state of most actual enterprise data, and even that leaves out what is happening in the social media domains, where the innovation, customer feedback and process insights are often occurring.

All those hard drives, all those iPods, all those memory sticks. All the metadata of what your workers do via the web -- out in the murky clouds -- it is all out of your reach. This is already the case, and it will not be a genie you can back in the bottle.

And the choices for retrenchment? Close off your workers from access to web search? Deny them access to your suppliers' portals? Erect firewalls that separate your customers, clients, and prospects from your own sales, marketing, and fulfillment providers? YOU CAN'T MOVE AT INTERNET TIME WITHOUT BEING ON THE INTERNET.

In other words, the ways we treat data today is unnatural. And pretending otherwise is unsustainable. There is a huge productivity opportunity for those that can re-think data, from concept to execution, that can exploit the gathering clouds. This requires radical rethinking, I'm pretty sure, though I can't say I know what the new data landscape will look like.

It will take a social capital-level ecology of complex systems in a pattern of ongoing churn for the answers to arise and quickly become outdated. There are deep and fundamental disruptions under way in media and software now that will force the hand on data. If you don't use what's free from the web and clouds, and your competition does, then what?

So what comes next? There's this murky middle muddle now between public and private clouds, SOA, independently located data, master data, and tools and development. If Google App Engine, Amazon Web Services and the near-certain follow-ons from Microsoft/Yahoo, HP, Oracle, IBM, and perhaps a handful of other major players have even a minor impact -- data conceptually is deeply changed. The cloud providers will give way the tools, single sign-on ID management, runtime, storage, and they will probably let you keep the data that you think is your data.

What enterprises will no longer have is the control of the market data from the public clouds about the users, the groups, their behaviors, attention, their demographics, buying patterns -- all the service on-ramps and off-ramps to your enterprise's revenue growth. Private clouds may not alone be enough to reach mass or long-tail audiences. Will your database of users, your email blast list, your CRM data be as good as Google's, or Microsoft's? Can you sell and market your goods and services without using online ads? Who will sell them to you?

If you're thinking of data as associated to internal applications, as the read-write store for ERP activities, or as the list of your sales leads and prospects in CRM -- you may want to think again. Your data and the clouds' data will need to work together, and perhaps those that bite the bullet and leverage the public cloud's to the hilt will have a huge advantage over those that do not.

Take a hard look at the diagram on this blog by Dan Farber, called "Google's Vision." Salesforce.com and Google think of data quite a bit differently than you do.

And just as you are in cozy partnership (like you have a choice) with your ERP or SOA vendors, enterprises and businesses of almost all sizes will be in partnership soon with the clouds. The shared and protected data alike will be scattered about, too complex for closed marts and masters. The cloud that can manage data in way that allows both user-level and process-level access, with granular permissioning -- and allows CXOs to feel good about it all -- gets the gold ring. The cloud business is a 50-year business.

How should your data be treated in this new cloud era?