Integration infrastructure approaches adjust to new world of SaaS and shared services

Read a full transcript of the discussion. Listen to the podcast. Sponsor: Cape Clear Software.

Change is afoot for the role and requirements of integration for modern software-as-a-service (SaaS) providers and enterprises adopting shared services models. Reuse is becoming an important issue, as are patterns of automation.

The notion of reuse of integration -- with added emphasis on integration as a service -- has prompted a different approach to integration infrastructure. The new demand is driven by ecologies of services, some from the Web "cloud," as well as the need to efficiently scale the delivery of services and applications composed of many disparate component services.

Integrations require reusable patterns, high performance, as well as many different means of access from clients. As a result Cape Clear Software has this week unveiled a new major version of its enterprise service bus (ESB), Cape Clear 7.5, with an emphasis on:

• A new graphical editor, the SOA Assembly Editor, an Eclipse-based tool to graphically clip together elements of integrations.

• Multi-tenanting additions to the ESB that allow segmentation of integrations, data, and reporting, as well as segmenting use and reuse of integrations on reporting and management of integrations based on the identities of inbound customers, clients, or businesses.

• A Business Process Execution Language (BPEL) management system with tools to monitor transactions, and repair transactions when they fail, to allow for rebuilding previous business information and ensure transactional integrity in running and maintaining large enterprise-class BPEL deployments.
  

To help better understand the new landscape for integration models, I recently moderated a sponsored podcast discussion with Phil Wainewright, an independent consultant, director of Procullux Ventures, and fellow ZDNet SaaS blogger, as well as Annrai O’Toole, CEO of Cape Clear Software.

Here are some excerpts:

... We're getting more sophisticated about SaaS, because it's being taken on board in a whole range of areas within the enterprise, and people want to do integration.

There are two forms of integration coming to the fore. The first is where data needs to be exchanged with legacy applications within the enterprise. The second form of integration that we see -- not at the moment, but it’s increasingly going to be an issue -- is where people want to integrate between different services coming in from the cloud. It’s a topic that’s familiar when we talk about mashups, fairly simple integrations of services that are done at the browser level. In the enterprise space, people tend to talk about composite applications, and it seems to be more difficult when you are dealing with a range of data sources that have to be combined.

People have realized that if you're doing integration to each separate service that's out there, then you're creating the same point-to-point spaghetti that people were trying to get away from by moving to this new IT paradigm. People are starting to think that there's a better way of doing this. If there's a better way of delivering the software, then there ought to be a better way of integrating it together as well.

Therefore, they realize that if we share the integration, rather than building it from scratch each time, we can bring into the integration field some of the benefits that we see with the shared-services architecture or SaaS. ... The new generation of SaaS providers, are really talking about a shared infrastructure, where the application is configured and tailored to the needs of individual customers. In a way, they’re segmented off from the way the infrastructure works underneath.

When you build an integration, you always end up having to customize it in some way for different customers. Customers will have different data formats. They’ll want to access it slightly differently. Some people will want to talk to it over SOAP. Some won't, and they’ll want to use something like REST. Or they might be going backwards and are only able to send it FTP drops, or something like that.

Multi-tenanting is one solution to the problem. The other is what we call multi-channel, which is the ability to have an integration, and make it available with different security policies, different transports, and different transformations going in and out.

A combination of multi-tenanting and multi-channeling allows you to build integrations once, make them accessible to different users, and make them accessible in different ways for each of those different customers. It gives you the scalability and reuse you need to make this model viable.

One point worth bearing in mind here is that this problem is going to get solved, because the economic reality of it suggests that we must solve this. One, the payoff for getting it right is huge. Second, the whole model of SaaS won’t be successful, unless we skin the integration problem. We don’t want the world to be limited to just having Salesforce.com with its siloed application.

We want SaaS to be the generic solution for everybody. That’s the way the industry is going, and that can only happen by solving this problem. So, we’re having a good stab at it, and I'll just briefly address some of the things that I think enable us to do it now, as opposed to in the past. First, there is a standardization that’s taken place. A set of standards has been created around SOA, giving us the interoperability platform that makes it possible in a way that was never possible before. Second is an acceptance of this shared-services, hosted model.

Years ago, people would have laughed at you and said, "I’m going to trust all my customer data to a provider in the cloud?" But, they’re doing it happily because of the economics of it. The whole trend toward trusting people with outsourced offerings means that the people will be more likely to trust integrations out there, because a lot of the technology to do this has been around for quite some time.

In enterprises you’re seeing this big move to virtualization and shared services. They’re saying, "Why are we having development teams build integration in all these branch offices at all these locations around the world? It’s extremely wasteful. It's a lot of skill that we've got to push out, and there are a lot of things that go wrong with these. Can't we consolidate all of those into a centralized data center? We’ll host those integrations for those individual business units or those at departments, but we'll do it here. We’ve got all the expertise in one place."

Those guys are delighted, because at the individual local level they don’t maintain all the costs and all the complexity of dealing with all the issues. It’s hosted out in their internal cloud. We haven't seen enough data points on that, but this hosted integration model can work. We’ve got it working for pure entities in SaaS companies like Workday, and we’ve got it working for a number of large enterprises. There is enough evidence for us to believe that this is really going to be the way forward for everybody in the industry.

Read a full transcript of the discussion. Listen to the podcast. Sponsor: Cape Clear Software.

Integrien deepens analytics, betters interoperability and userability in Q4's Alive 6.0 release

Read a full transcript of the discussion. Listen to the podcast. Sponsor: Integrien Corp.

The movement of IT and systems management to the end-to-end business service value level has been a long time in coming. Yet the need has never been higher. Enterprises and on-demand application providers alike need to predict how systems will behave under a variety of conditions.

Rather than losing control to ever-increasing complexity -- and gaining less and less insight into the root causes of problematic applications and services -- operators must gain the ability to predict and prevent threats to the performance of their applications and services. Firefighting against applications performance degradation in a dynamic service-oriented architecture (SOA) just won't cut it.

By adding real-time analytics to their systems management practices, IT operators can determine the normal state of how systems should be performing. Then, by measuring the characteristics of systems under many conditions over time, administrators can gain predictive insights into their entire operations, based on a business services-level of performance and demand. They can stay ahead of complexity, and therefore contain the costs of ongoing high-performance applications delivery.

I recently had a podcast discussion with Mazda Marvasti, the CTO of Integrien Corp., on managing complexity by leveraging probabilistic systems management and remediation. I learned that Integrien's Alive suite uses probabilistic analysis to predict IT systems problems before costly applications outages. Furthermore, I received some details on the next Alive 6.0 release in Q4 of this year.

Here are some excerpts:

Can you give us some sense of the direction that the major new offerings within the Alive product set will take?

Basically, we have three pillars that the product is based on. First is usability. That's a particular pet peeve of mine. I didn't find any of the applications out there very usable. We have spent a lot of time working with customers and working with different operations groups. ... The second piece is interoperability. The majority of the organizations that we go to already have a whole bunch of systems, whether it be data collection systems, event management systems, or configuration management databases, etc.

Our product absolutely needs to leverage those investments -- and they are leveragable. But even those investments in their silos don’t produce as much benefit to the customer as a product like ours going in there and utilizing all of that data that they have in there, and bringing out the information that’s locked within it.

The third piece is analytics. What we have in the product coming out is scalability to 100,000 servers. We've kind of gone wild on the scalability side, because we are designing for the future. Nobody that I know of right now has that kind of a scale, except maybe Google, but theirs' is basically the same thing replicated thousands of times over, which is different than the enterprises we deal with, like banks or health-care organizations.

A single four-processor Xeon box, with Alive installed on it, can run real-time analytics for up to 100,000 devices. That’s the level of scale we're talking about. In terms of analytics, we've got three new pieces coming out, and basically every event we send out is a predictive event. It’s going to tell you this event occurred, and then this other set of events have a certain probability within a certain timeframe to occur.

Not only that, but then we can match it to what we call our "finger printing." Our finger printing is a pattern-matching technology that allows us to look at patterns of events and formulate a particular problem. It indicates particular problems and those become the predictive alerts to other problems.

Now, with SOA and virtualization moving into application-development and data-center automation, there is a tremendous amount of complexity in the operations arena. You can’t have the people who used to have the "tribal knowledge" in their head determining where the problems are coming from or what the issues are.

The problems and the complexity have gone beyond the capability of people just sitting there in front of screens of data, trying to make sense out of it. So, as we gained efficiency from application development, we need consistency of performance and availability, but all of this added to the complexity of managing the data center.

That’s how the evolution of the data center went from being totally deterministic, meaning that you knew every variable, could measure it, and had very specific rules telling you if certain things happened, and what they were and what they meant -- all the way to a non-deterministic era, which we are in right now.

Now, you can't possibly know all the variables, and the rules that you come up with today may be invalid tomorrow, all just because of change that has gone on in your environment. So, you cannot use the same techniques that you used 10 or 15 years ago to manage your operations today. Yet that’s what the current tools are doing. They are just more of the same, and that’s not meeting the requirements of the operations center anymore.

I’ve been working on these types of problems for the past 18 years. Since graduate school, I’ve been analyzing data extraction of information from disparate data. I went to work for Ford and General Motors -- really large environments. Back then, it was client-servers and how those environments were being managed. I could see the impending complexity, because I saw the level of pressure that there was on application developers to develop more reusable code and to develop faster with higher quality.

The run book is missing that information. The run book only has the information on how to clean it up after an accident happens.

That’s the missing piece in the operations arena. Part of the challenge for our company is getting the operations folks to start thinking in a different fashion. You can do it a little at a time. It doesn’t have to be a complete shift in one fell swoop, but it does require that change in mentality. Now that I am actually forewarned about something, how do I prevent it, as opposed to cleaning up after it happens.

Read a full transcript of the discussion. Listen to the podcast. Sponsor: Integrien Corp.

Before we know who owns the SOA business case, how about simple business processes?

There's a good article on "owning" the business case for SOA on SearchWebServices.com. Some of my most respected analysts are quoted.

But is the question posed a relevant one? While making the business case for SOA is and will be a fascinating topic for some time, we may be jumping the gun.

From where I sit, just about everyone that has a strategic role in IT and business decisions at an enterprise has an "ownership" stake in SOA. It's that pervasive. The COO may be the best person under many current organizational charts to see all the moving SOA parts.

Yet buy-in and inclusiveness -- both wide and deep -- for SOA are essential, so it can't really fall to any one person. Assigning "business value" ownership is too abstract, really, for real-world companies to begin using it and embracing SOA. SOA is ubiquitous in its effects. The positioning of SOA as an abstraction is holding back its embrace and adoption.

So let's look at more practical questions on SOA and business value, before we go shooting for the moon. Sadly, in even the most progressive enterprises, the ownership of a single business process is ambiguous. Organizations have been ceated for decades based on the notion of decentralization -- which is just another way of breaking up complexity into small chunks and assigning responsibility for the chunks, often at the expense of minding the whole. Very few individuals or teams are defined or incentivized to manage an entire business process. Yet this an essential stepping stone to SOA, and to eventually making the business case for SOA.

We see attempts to proffer SOA from the top down, with even less emphasis on adoption from the bottom up. What I'm saying is also, and perhaps predominantly, build it from the middle out. Create the new middle for SOA at the business process level, and then evangelize it in any which way.

In effect, SOA and its foundational core, business processes, are fighting back against the long-term tide of decentralization and IT specialization. SOA says you can now make the chucks of discrete IT resources relate far better, so why not begin to look at an entire process and work to make it more efficient, and more flexible? Why not extract the best of specialization and improve, refine and reuse the parts best in the context of general business -requirements whole? See the forest and the trees. Make better business decisions -- operationally and strategically -- as a result.

As Dr. Paul Brown points out in a book I recently helped review via a sponsored podcast, Succeeding with SOA: Realizing Business Value Through Total Architecture (Addison-Wesley, April 2007), the business process is the right level to assign "ownership." Now.

When an analyst or architect -- as well as their teams -- begin to see themselves as managing and evangelizing on a business process level, then SOA can begin to make strides as a concept and methodology more broadly. To try and inject SOA into a company broadly, then discretely is putting the cart in front of the horse. Better yet to re-arrange all the horses and carts based on the right trips for the right loads, making it easier to change horses and carts as needed.

I like the idea of cross-functional teams (horses, carts, drivers, and caravans) created that serve a business process lifecycle. These would be pods (perhaps virtual in nature) of tightly-coordinated people with the right mix of skills and experience -- specific and general, technical and business-oriented, able to communicate as a team on many levels.

Like the Ray Bradbury book, Fahrenheit 451, where individuals learn and carry on whole books in their memories as a way to preserve the books and their knowledge, business process pods would retain and refine the essence of a business process and care for it and extol its virtues throughout an enterprise. They would cross all the chasms across the constituent services but at the higher business value level.

We've heard talk of a "T" person from SOA evangelists at IBM, whereby the horizontal bar in the "T" represents business acumen, and the vertical bar represents technical depth. But I like the idea of the cross-functional pod better -- a team of, by, and for the business process.

The ownership of a business process (never mind SOA) is too much for one person. A multi-talented team can provide the wetware and organizational dynamism to get SOA started on a practical, middle level -- that of a business process as a productivity entity. This step is what's needed before we start assigning ownership for the business case for SOA.

Developing enterprise applications for mobile devices remains way too hard

A logjam exists between developers and their ability to productively deliver enterprise applications and data to mobile devices, such as cell phones, PDAs, and so-called converged devices like the Apple iPhone.

The logjam is complexity and too many obnoxious variables. To develop applications that reach even a small number of major handset environments means big-time custom plumbing, from the various data sources, to the mixture of networks, to the choices on synchronization, to the various security needs, to the many user interfaces and mobile client operating systems. Managing all these variables requires a high degree of skill across many different skill sets. There are not many developers that fit this bill in your average enterprise.

And this all means a lot of time and money is required to bring just a few basic applications to just a few basic mobile clients. No wonder enterprise mobility stubbornly remains below the radar for IT leadership. Mobile remains relegated to the crowded back burner of IT imperatives.

And given all the variables and high degree of required customization, few ISVs have emerged to try and make a living at producing mass-market mobile applications. The subsequent lack of killer applications, other than standards-based email and text messaging, reduces the appetite to take on the infrastructure complexity for taking the corporate datacenter out to the mobile client across commercial mobile networks.

Also impacting the complexity is the diversity in how mobile devices work from geographic and regulatory market to market. It's nearly impossible to envision a global approach to mobile wireless computing, as we've seen for desktop and web computing. Designing for one mobile market does not give you much of a leg-up in reaching many others.

The "inclusive platform" approach of dictating the exact device and/or runtime environment up and down the stack that mobile applications play in is itself stifling given the inability to take advantage of the low-cost devices and services available via commercial mobile service carriers. What's more, aligning the back-end and front-end infrastructure does not necessarily align with how the mobile telecommunications carriers and handset operators, well ... operate. They don't like the idea of losing control of what their clients do on their networks.

Clearly, wireless handheld delivery of enterprise data and applications has yet to reach its untapped and vast potential.

Microsoft for years has been grappling with these issues, with many fits and starts. There have been some impressive successes with Windows Mobile, but Microsoft has by no means sewn up the field of mobile enterprise applications design and delivery. Microsoft with its Windows Mobile approach has not yet achieved a critical mass for how business applications and data can be driven out to a field-based workforce. And it's likely that today's widely heterogeneous environment for end points and devices will be with us -- at least in the U.S. -- for a long time to come.

These seemingly intractable roadblocks to wider mobile business use are one reason we're seeing what amounts to appliances for the client devices. The Apple iPhone, which launched with great fanfare in late June, is a prime example. Apple has fused hardware, software and applications -- as well as a few critical APIs -- and has picked one opening-inning carrier, AT&T, as well as Wi-Fi generally, for connectivity. And that combination of attributes, along with what should soon be more APIs, makes the converged device/appliance approach not just a consumer affair -- it begins to help reduce the complexity for enterprise mobility too.

Microsoft in the summer said that it likes the idea of such fused mobile clients -- the appliance on the client -- so much that it has hinted it will produce such integrated devices too. Rumors persists that Google also has its sights on a mobile handheld appliance of some sort. Things are clearly heating up.

These nifty clients should go quite a way to making enterprise mobility far easier by allowing developers to exploit them with fewer interface and connectivity variables to manage. But the converged client still needs a back-end or middleware counterpart to help coordinate an enterprise's data, logic, and security needs. Sybase is hard at work on what may prove to be a game changer for such enterprise mobility middleware -- especially when coupled with a converged device such as the iPhone.

Sybase has bet its future growth on mobility. And while the Dublin, Calif. company has not yet announced the details of its full stack, the Sybase vision makes a lot of sense. Expect in the next year to see what amounts to a distributed middleware system approach to enterprise mobility from Sybase that may break the current logjam in enterprise mobile development and deployment. If Sybase can shake up the enterprise mobility infrastructure industry -- and partner effectively with the likes of Apple's iPhone -- then Microsoft's response will need to be swift and significant.

Dare I say it? What the heck: Look to 2008 to be the year that enterprise mobility finally gets some legs.

Survey uncovers heightening reliance on search across business purchasing

Listen to the podcast. Or read a full transcript. Sponsor: ZoomInfo.

It seems that businesses, whether they're small or global 2000 concerns, are buying more supplies using search at some point in the B2B procurement process. Some people begin and end a procurement journey with search. They actually buy the products through a strictly search-dependent process.



Yet many still use a combination of word-of-mouth, search, and traditional information gathering to guide them to the best deals on the most goods.

To find out just how much B2B buying behaviors are shifting, Enquiro Search Solutions conducted a survey earlier in 2007. They found that online search was consistently employed throughout the entire buying process, from awareness right through to purchase.

There’s still a lot of back and forth: Offline factors influence online activity, and vice-versa, for a merging of the online and the offline worlds. In an audio podcast discussion, as well as the accompanying BriefingsDirect multi-media video-podcast, I helped plumb the depths of Enquiro's findings and then vetted them through the experiences of B2B search engine ZoomInfo.

Join Gord Hotchkiss, President and CEO of Enquiro, and Bryan Burdick, COO of ZoomInfo, with moderation by myself, Dana Gardner, for a deep dive on B2B search trends and analysis.

Here are some excerpts:

We did the original survey in 2004 and, at the time, there wasn't a lot of research out there about search in general, even on the consumer side. There was virtually nothing on the B2B side. The first survey ... certainly proved that search was important. We found that online activity, in particular that connected with search activity, was consistent in a large percentage of purchases. In 2007, we added more insight to the methodology. We wanted to understand the different roles that are typical in B2B purchases -- economic buyers versus technical buyers versus user buyers. We also wanted to get more understanding of the different phases of the buying cycle.

As far as the main takeaways from the study, obviously online activity is more important than ever. In fact, we asked respondents to indicate from a list of over 30 influencers what was most important to them in making the purchase decision. Online factors, such as interaction with the vendor Website and interaction with the search engine were right up there with the traditional winner, word of mouth. What we see is a real link between those and looking for objective information and specific detail.

We did notice an evolution of behavior as you move through the funnel, and the nature of the interactions with the different online resources changes how you navigate to them and how you go to different sites for information. But, online research was consistent through the entire process, from awareness right through to purchase. There’s a lot of back and forth. ... We saw a merging of the online and the offline worlds in making these decisions and trying to come to what’s the right decision for your company or what’s the right product or service.

We just found increased reliance on online to do that research. When we say "increased reliance," we're probably talking 10 percentage points up over the three years. So, if 65 percent of the people were doing it in 2004, 75 percent of the people are doing it now. That’s primarily where we saw the trends going.

When we looked at the different phases of the buying cycle, it starts with awareness. You become aware that you need something. There was a high percentage of people -- in the high 60-percent range -- who said, "Once I become aware that I need something, the first place I'm going to go is the search engine to start looking for it." A lot of that traffic is going to end up on Google. It was the overwhelming choice among general search engines for B2B buyers.

But, as you move through the process, you start doing what we call a "landscape search." The first search is to get the lay of the land to figure out the information sites that have the information you are looking for. Who are the main vendors playing in this space? Where are the best bets to go and get more information to help make this purchase decision?

So, those first searches tend to be fairly generic -- shorter key phrases -- just to get the lay of the land to figure out where to go. As you progress, search tends to become more of a navigational shortcut, and we’ve seen this activity increase over the last two to three years. Increasingly, we're using search engines to get us from point A to point B online.

We also wanted to get a retroactive view of a successful transaction. So, in the second part of the survey, we asked them to recall a transaction they had made in the past 12 months. We wanted to see whether that initial search led to a successful purchase down the road, and, at the end of the road, how the different factors influenced them. So, we actually approached them from a couple of different angles.

Now, 85 percent of these people say they're using online search for some aspect of this purchasing process. It strikes me that this involves trillions of dollars worth of goods. These are big companies and, in some cases, buying lots of goods at over a hundred thousand dollars a whack. Do you concur that we're talking about trillions of dollars of B2B goods now being impacted significantly by the search process?

Absolutely. The importance of this is maybe the most mind-numbing fact to contemplate. Traditionally, the B2B space has been a little slow to move into the search arena. Traditionally, in the search arena, the big advertisers tend to be travel or financial products. B2B is just starting to understand how integral search is to all this activity. When you think of the nature of the B2B purchase, risk avoidance is a huge issue. You want to make sure that whatever decisions you make are well-researched and well-considered purchases. That naturally leads to a lot of online interaction.

The business information search is a primary factor driving [ZoomInfo's] growth. Our company right now is growing on two fronts. One is our traditional paid-search model, where we have subscription services focused on people information that is targeted at salespeople and recruiters as a source for candidates and prospects.

The more rapidly growing piece of our business is the advertising-driven business information search engine, which I think is a really interesting trend related to the concept you guys were just talking about. Not only does the B2B advertiser spend lots of money today trying to reach out, but the B2B searcher has new tools, services, and capabilities that provide a richer, better, more efficient search than they’ve had through the traditional search engines.

Everybody needs to be focused on search. I can’t see an exception. You mentioned the percentage that said they would go online. We segmented out the group that didn’t indicate they go online to see what was unique about them. The only thing unique about them was their age. They tended to be older buyers and tended to be with smaller organizations, where the CEO was more actively involved in the purchase decision. That was really the only variants we saw. If it’s a generational thing, then obviously that percentage is going to get smaller every year.

... From a vertical business information search perspective, that we’re really in the first inning here. A lot of interesting trends and enhancements are going to be coming down the road. One in particular that may have an influence in the next year or two is the community aspect within the search. ... I think that you’ll start to see a marriage of, not only B2B search, but also online community and a factoring into that whole process. Then, who knows where we’ll go from there? ... The word of community.

Listen to the podcast. Or read a full transcript. Sponsor: ZoomInfo.

Book review discussion: 'Total Architecture' elevates SOA to its business benefits potential

Listen to the podcast. Or read a full transcript of the discussion. Sponsor: TIBCO Software.

The impact that Services Oriented Architecture (SOA) has on an organization is deep and wide. The changes required to bring SOA to fruition and make it as productive as possible affect the way both IT leaders and business managers think and operate.

To provide a solid foundation on how SOA impacts a business, Dr. Paul Brown, a principal software architect at TIBCO Software, recently wrote "Succeeding with SOA: Realizing Business Value Through Total Architecture." The book emphasizes a "Total Architecture" perspective for SOA and advocates how business processes are the real focus for enterprises to prosper.

Many activities that used to be done manually either completely or partially now ingrained in IT systems. SOA helps elevate those activities into loosely coupled services with standard interfaces so they can be used and reused flexibly within business processes, while leveraging the underlying IT assets and automation benefits.

Brown calls on businesses to define, fully grasp, control, manage and adjust the processes as rudders that steer overall business agility, powered by SOA methods. However, enterprise business processes and enterprise IT systems are so intertwined that you can’t really talk about designing one without designing the other -- hence the need for "Total Architecture."

In this podcast, Dr. Brown is interviewed about his book and the concepts of Total Architecture by enterprise architect Todd Biske, with moderation by myself, Dana Gardner, principal analyst at Interarbor Solutions. I do hope that people take a look at this book. I think it has a lot to offer.

Here are some excerpts:

The structure and organization of IT is very often not completely aligned with the business. The care and feeding of the IT infrastructure has become a focal point of a lot of the IT investment. What we’ve lost is the connection between what’s going on in IT and what’s going on in the business.

The structure of our IT solutions are difficult to map onto the structure of the business processes. It’s that realignment that we’re trying to address with SOA. Ideally, the services that we’re building -- that have technical implementations -- are building blocks of business processes.

The real business value comes out of having a portfolio of business functionality in the form of services that you can quickly reorganize and re-orchestrate to build new business processes or to modify your existing processes. That’s where the big business win comes in.

Dr. Brown points out that a lot of times IT may be misaligned and not be able to see the business processes that are spanning the silos. It may also be the case that the business isn’t either. So, you’re really setting up challenges, when you’re trying to view things from more of an enterprise perspective.

... The pressures of globalization make viewing your entire enterprise and your business as a whole even more important. It’s hard to continue to operate in a niche and continue to sustain that organization for many years, whether it’s from the IT aspect or the business aspect.

It’s distressing, when I go into a company, how infrequently I find anybody who can articulate what the end-to-end business process is that produces the key results or can tell me what the entire order-to-cash cycle looks like. There are lots of experts for fragments of it. The IT focus has traditionally been on fragments of the processes.

The pressure these days is to improve the overall business process, the response time to the customers and partners, and the overall quality of what’s going through. So we need this focus on end-to-end business process and the focus on the end-to-end system interaction that helps bring that business process to life.

What we need to introduce into the picture is more thinking in terms of how these different silos play together to make the business work. That has to happen on both the business side, looking at business processes, and the technology side, looking at systems.

... We need a proactive enterprise architecture group that’s willing to roll up its sleeves and get its hands dirty, touching the individual projects that are going on. They really need to coordinate their work, so that we don’t end up with the chaos that we’re in today, and so that we have pieces that elegantly fit together and can be rearranged to achieve new business goals.

When you design a business process, you’re making assumptions about what systems are going to do and what people are going to do. You can’t validate the assumptions until you actually start doing the technical design. You need to have a systems architect involved when you are planning business processes.

Conversely, you can’t really make modifications in the system without inadvertently changing the business processes. So it doesn’t make sense to separate these activities. You really have to treat them as if you're architecting the business, which consists of both business processes and systems.

One of the things that was impactful for me was the notion that SOA can be a catalyst to thinking about things differently and bridging this business-IT chasm. And also that SOA can be a precursor to a whole new conversation about how to think about businesses from this perspective of "Total Architecture."

... Your business processes and your systems are intertwined, and the design of one affects the other. That’s reality. You can choose to ignore your business processes and let them evolve by accident, as Todd was talking about earlier. That’s how we got in the bind we’re in right now. To a large extent, we’ve been focused on individual profit centers and driving down cost in individual functions. We’ve lost sight of the end-to-end business processes.

The resurgence of interest in customer loyalty, customer service, and all of that is really a reflection that, as a business, we need to stand back and look at what we’re doing to our customers and our partners. The investments we’re making and the individual activities in the business processes are fine, but in order to remain competitive we need to stand back and take the holistic view. The holistic view simply means you need to understand what that business process looks like before you can improve it.

I’ve seen horror stories of people who have tried to improve a business process by examining a small portion and making some changes, without realizing that the changes that they’re making are having an adverse impact on the bigger business process. So while they thought they were improving things -- they actually made them worse. You can’t live with that.

Listen to the podcast. Or read a full transcript of the discussion. Sponsor: TIBCO Software.

SaaS adds up better with eXpresso's hosted solution for Excel collaboration

With over 150 million business users worldwide, Microsoft's Excel has become a de facto standard for vast sets of business information and collaboration points.

Problems arise, however, when users and managers try to share spreadsheets and coordinate updated internal Excel information, sometimes from hundreds of far-flung users. Those seeking ease in distributing the contents of spreadsheets often bemoan the closed and brittle nature of "spreadmarts" -- the burgeoning assemblages of spreadsheets, usually amid multiple versions of each.

eXpresso Corp., Menlo Park, Calif., has introduced what it says is a unique hosted solution that provides broad functionality while eliminating the cost and need for the IT department's involvement for even minor Excel support, access and customization.

Using native Excel components, eXpresso's product allows the spreadsheet owner to invite other users to view, edit, and update a spreadsheet, while retaining control over the permissions that each user has, right down to the cell level -- something current online collaboration tools, such as Google spreadsheets, don't currently offer.

Because it's a SaaS, hosted system, users don't need to download software or involve IT departments, something that would be required with such solutions as Microsoft Office SharePoint Server. The hosting also allows collaboration outside the firewall, something that can present problems with in-house collaboration solutions.

A bonus comes with eXpresso's ability to track changes, provide an audit trail of who changed what and when, and even to allow the spreadsheet owner to roll back the data to a previous version on an individual user basis. Email alerts can notify users when changes are made.

Currently a free offering, in October eXpresso will begin charging $15 a month per user for advanced features, while the core functions will remain free.

As more features and utilities, rather than the usual whole applications, become available via SaaS, the productivity and acceptance of on-demand services grows and propels even more entrants to the ecology.

eXpresso provides a great example of potentially high productivity at high convenience and low cost. We should expect to see a lot more services and widgets like this. I wonder if they could use FaceBook -- or other social networks -- to distribute knowledge and even the use of this service?

More obvious misgivings about Microsoft and SOA

InfoWorld blogger and MuleSource CEO Dave Rosenberg has some thoughts on Microsoft and SOA in light of, and in advance of, the recent spate of BizTalk announcements and partnerships. Like myself, he takes exception to Microsoft's claims of SOA support and affinity.

Here are some excerpts from an interview Rosenberg did with ITBusinessEdge:

Question: So. You're among those who feel Microsoft doesn't "get" SOA? Why do you say so? Do you think they don't get it -- or don't want to get it?

Rosenberg: I would say that not only does Microsoft not get SOA, they purposely are trying to usurp the whole concept for their nefarious doings. Microsoft offers nothing in the way of architectural development tools or infrastructure that supports the ideas behind an SOA. The Microsoft architecture of .NET is not designed to be service-enabled and when you try to service-enabled it yourself -- when you try to build the infrastructure so you can take advantage of reuse, you find yourself in a very customized system having defeated the purpose.

If you look at Microsoft infrastructure, it's all about trying to lock people into their hegemony and SOA is all about giving control to the user.

Question: Did you see Microsoft's 100-something page thesis on SOA, titled "SOA in the Real World?"

Rosenberg: I can't tell you I've sit there and poured over every word. It's not unlike their "Get the Facts" anti-Linux campaign -- it's simultaneously interesting and full of crap at the same time.

What's interesting to me about Microsoft's approach is the obvious thing to do with SOA is to say, "Of course we have a strategy -- here's what you do now and here's what we'll do in the future." What should've been very easy for them to say, "Yes, we'll be a part of this and we want to start to think about our way of doing [SOA]" -- that would be acceptable. Instead they take this bizarre approach.

There's no clear answer from Microsoft on what their vision for SOA is or how their products or things you'd buy from them would participate in a SOA. For that matter, there's nothing from Microsoft that would say to someone, "I should use these products for my SOA."

All of the sudden .NET went from being a language, to an application framework, to being a "Windows platform" again.

Developers are quick to shun things that they don't trust and I think Microsoft sets the tone for how their development community thinks about larger scale concepts and so far they've succeeded in making SOA confusing.

Question: So, is Microsoft's talk about SOA a barrier to its acceptance among developers?

Rosenberg: From what I can tell, they're not doing themselves any favors. The whole sort of wait-and-see approach is not great. What's interesting is developers do eat that Microsoft dogfood pretty fiercely. They wait for Microsoft before they make choices.

It's a challenge for architects in terms of being flexible and having agility in their work. If you look at IBM or BEA, it's very clear what their vision of SOA is. With Microsoft it's just not clear.

I think some of that is related to the fact that .NET was not built to be a SOA or a services platform. They don't want a heterogeneous environment, they want the entire Microsoft suite everywhere. For instance, a couple of months ago they were calling it service-oriented infrastructure. What is the rationale with not going with the industry standard terms?

Before I started this company [MuleSource], I was the CIO of financial services firm. Initially we had an all LAMP and Java infrastructure. Prior to my getting there, they outsourced the development of a .NET application. We went from having a highly scalable, flexible infrastructure that was able to consume data and services across the enterprise and added a .NET application into the infrastructure -- it was a complete and utter silo.

It was built around this framework that was not meant to go in a services direction, and we basically had to adjust everything else in the enterprise for this one application that couldn't align itself with the rest of the business.

And that is the unfortunate side of what .NET has done to development. Conceptually, Java guys are more likely to understand the SOA model than the .NET guys are -- developers are not trained to think about other applications in .NET. They aren't trained to think about wanting to consume or expose a service to another application. That has so far been a barrier to using .NET as a platform to do SOA.

For better or worse, this development style is what .NET developers live and breathe. If Microsoft doesn't have a component, it doesn't exist. But in the real world, you still need to solve that problem.

How can a company that in-the-know be so clueless about as an important concept as this? This is the world's largest technology company and it's proven to be completely impotent and useless with SOA so far.

Question: So, do you think this is a sort of marketing ploy by Microsoft or just because they didn't anticipate that SOA would take off?

Rosenberg: I think it's a bit of both. What I'd tell you is that they should embrace SOA. Open source, you can see why they don't embrace it -- but SOA, they should say it's awesome and encourage it. It doesn't make a lot of sense.

In the real world, we see people who have similar situations to the one I described, with one or two .NET applications that they want to get on the [enterprise service bus (ESB)]. Does Microsoft give you an obvious answer? Kind of, but not really.

My take is that inside of Microsoft its aggressor A-types are all about dissing SOA and promoting .NET ad nauseam. At the same time the Microserfs and developers must understand the inevitability of SOA for at last a portion of the most advanced and innovative enterprises' and service providers' architectures.

And so, as the world turns toward SOA, Microsoft will fight quietly inside of itself about what it really is as a company -- a partner to its customers, or a parasite on the hide of productivity.

Ultimately the marketplace will determine Microsoft's end-game role. If there's an advantage to SOA for those that embrace it broadly and effectively -- and I believe without question that there is -- then there will be a penalty for those that do not embrace SOA principles. This will become apparent first as SaaS and hosted, on-demand applications providers.

For those IT shops that throw their infrastructure fates to Microsoft's software development and business development competencies (where the latter is the strength), they may encounter cost and agility disadvantages.

If I and Dave Rosenberg are wrong on SOA benefits and n Microsoft's lack of general support for SOA, then the more purely .NET shops should demonstrate market strength via lower TCO and greater business agility over time. The .NET-based businesses should play better amid complex, global business ecologies, and be able to take advantage of mixed sourcing across a waterfront of available services.

Under the dictates of comparative advantage, .NET and its Microsoft-oriented progeny should not create greater value and higher productivity for its customers than more general alternatives, such as open SOA. A choice of the best service for the job will dominate a choice over only the .NET service available.

I'll be waiting and watching, though my risk is an observer is much lower than the architects placing their bets in the coming years. The ante, incidentally, is the very survival of their companies as we enter an "flattened world" era of increased globalization, lower barriers to entry, open trade, and lightening fast market disrupters.

For those with an inclination to hedging bets -- banking on general SOA while supporting service-enabled .NET might be more secure and auspicious than banking on .NET while trying to support SOA objectives from a comparative disadvantage.