I've said it before: Microsoft should buy BEA

The dark clouds over BEA are deepening with news that activist Carl Icahn is buying up tangible influence in the software vendor (probably with the idea of a sale). BEA faces lingering unknowns on its financial statements -- not to mention entrenched concerns about sales, growth and its ability to compete against Microsoft, IBM and myriad open source alternatives.

The Icahn news bodes badly for a healthy, wealthy and independent BEA. Whether through an outright sale or false-value run-up of BEA's value and then abandonment at a false peak (remember Time Warner and Motorola), the status quo won't hold.

A few years ago when BEA was at another crossroads (like, most of the technical management tier walked out), I suggested that Microsoft should buy them. It was a novel declaration, and some snickered. Others found surprising logic in the argument but for the Microsoft religion thing around Windows Everywhere.

I think the idea of a Microsoft-BEA mashup deserves merit more now than ever. And it has more to do with Microsoft than BEA, largely because Microsoft has changed with the times in the past several years while BEA has mostly stayed the course, just renaming theirs a SOA value while voraciously resisting the middleware moniker while selling predominantly ... middleware.

As we see Microsoft truly surrounded by vipers -- despite its huge strength, influence and tenacious installed base -- it recognizes that the effects of open source on software economics is not going away. Nor is IBM. And it also knows that advertising revenue -- even at the clip that Google assembles and attracts it -- will not make Microsoft live as well as it has the past 10 years for the next 10 years.

Microsoft is not in any where near the trouble that BEA is, but BEA's history may be worth studying -- for it could be a fast-forward version of Microsoft's.

Windows Everywhere used to make great sense (and cents), but the key to remaining an IT supplier growth engine for the global 2000 enterprises and legions of SMBs is not, alas, Windows Everywhere. IT departments everywhere have a different goal than to make Microsoft their Daddy. They need to continuously cut costs while providing better services, more interoperability, and to elevate agility to the level of malleable business processes. The divergence between Microsoft's business goals and the goals of its core customers is wide and growing.

IBM gets this. HP gets this. Oracle gets this. The global SIs get this. Microsoft could get it -- and perhaps solve it substantially -- if it bought BEA now and became a true enterprise IT solutions provider; if it took the best tools and a great framework and made them the overlay to developing and managing the best combination of mainstream IT architectures, platforms and frameworks.

What's happened in the past five years is that the development, ISV, and hosting communities have divvied up into separate camps around .NET and Java. Many need to try and play nice together, but it's not been easy. It's expensive. And not only is it technology that separates these camps -- they think differently. They see the role and use of open source much differently, and they view costs of IT differently, as in where should the costs be? An astute observer mentioned to me yesterday that .NET developers are from Venus and Java framework developers are from Mars. And so it goes.

But with a deep shift on the part of Microsoft toward really embracing and extending the Java side of the universe, then we're talking about dominating not just the .NET world, but perhaps the Java one, too. Even better, how about becoming the best way to leverage the mainframe, client/server, Web and SOA paradigms? How about building on that leverage with the best management, modeling, and integration for the best price? And you could bring the advertising, media, and consumer reach into the mix. Microsoft could become a one-stop shop, rather than a proponent of a re-hashed one-shop sop about Windows Everywhere.

The strategic risks are now such that an embrace of open source and Java-centric computing by Microsoft is at par with trying to foist Windows Everywhere on everyone (while eroding the boundaries slowly over time). Cut bait, I say, and fish anew -- and to hell with an instance on one rod over another.

We're really talking about cutting total costs and complexity, not purporting to cut them both while ignoring a major portion of the real world (and letting CIOs deal with it). BEA would allow Microsoft to redefine the distinct orbits of the .NET world and the distributed Java/open source tools world. The abstraction on where to seek lock-in nowadays is at the inflection point of RIAs, SOA and ALL the legacy.

A Microsoft-BEA combination would signal true neutrality as to underlying platform requirements -- while making the new differentiation point all about total management of IT costs and complexity. This is the value that IT buyers need in a partner, not just a vendor.

Analysts debate role of governance and 'total management' in the dawning era of SOA

Listen to the podcast. Read a full transcript of the discussion. Sponsor: Tidal Software.

I recently had the pleasure of participating in a live discussion (now a podcast) at the Harvard Club of Boston with Jason Bloomberg, managing partner at analyst firm ZapThink, on the current state and future outlook for governance and management in SOA environments.

Moderating the discussion was Martin Milani, chief technology officer at Tidal Software, which sponsored the luncheon event. We had dozens of enterprise IT executives from the Boston area in attendance, and they joined us in a questions and answer session after the presentations.

During the hour-long "debate" Jason and I explored how IT management will evolve in the world of service-based applications. The discussion delved into issues of new standards, how SOA demands that performance management and change management should augment and elevate the role of systems management, and on how the integrity of services delivery requires a deep and wide approach to "management in total" across a service's lifecycle.

A recent survey supports some of our conclusions: That more needs to be done to provide governance and management, and that the very definition of "management" requires re-evaluation in the era of SOA.

Here are some excerpts from the event:

The key thing to keep in mind about SOA in this context is that services are an abstraction. That is, they help to provide flexibility to the business, but they don’t actually simplify the underlying technology. Many architects are a bit surprised by this, that SOA doesn't make their jobs easier or make the job of IT any easier. If anything, it’s more complex.

There's more of a challenge for IT to meet the business requirements for flexible, agile, composable, and loosely coupled services. As a result, you have this need for the IT organization to rise to the challenge of services. This is especially true in the management area because the services essentially have to behave as advertised.

The implications for those dealing with applications is that you are going to service-enable those applications, decouple, and decompose them into essential core services, and then repurpose them by cross-compositing processes. What is that going to do to you, if you think you are going to go to firefighting mode when you have performance issues? It’s simply not going to work.

You need to rethink management and support, and you need to try to get proactive in how systems will be supported to head off performance issues and create insurance policies against blackouts, brownouts or other snafus. SOA is really a catalyst toward a different approach to the management and support of the services.

[SOA management] needs to be looked at not just as management of discrete parts, not just trees within the forest that each stand on their own -- but the forest itself. I'd like to see that get to the point where it becomes something that can be assimilated further than just the systems -- with the business objectives as well.

... You are going to want to tune how your applications and services are delivered, perhaps to live up to service level agreements, or perhaps so that you can give priority to certain data, application, or services streams over others. ... That’s going to require a different level of management. It’s really about leveraging the old, finding ways to assimilate and then put a more operator- or policy-driven -- perhaps even automated -- approach on top of it.

With SOA you need to gather information about your systems both deeply and broadly: deep and wide. You can already get a fire-hose of data from your systems, log files, and agent- and agentless-based approaches already on the market. You get a ton of data. It’s working with that data in the context of a horizontal business process that’s the hard part. ... If one aspect of a process goes down, that’s the weak link in that chain, the whole chain could be at disadvantage.

In the past, you might have one application down, but people could go off and do another task, because that mainframe would be back up at two o’clock. If your entire supply chain is disabled for a period of time, that’s a higher price to be paid. So, we're looking at a different level, and I don’t think we’ve seen the solution yet.

The value of IT here can be much greater. It can be an enabler, not a cost center. It can be the way in which not only is information relayed about what’s going on, but can determine what we want to happen. We want to change that supply chain. We want to change that distribution, recall these products, get a list of every single product and every serial number, and we want to relay that to our sales force.

That sounds straightforward, but if you try to do that with a lot of IT systems today, you’re going to find yourself up there with the equivalent of mimeograph and crayons, doing it by hand -- and that’s just not acceptable. So in the future, a company’s very existence could be at stake if they don’t have agility in these processes.

SOA is really not optional. Companies that don’t get this right will suffer the consequences. They will suffer lawsuits and suffer a competitive disadvantage. They are going to go out of business. This is an important thing to keep in mind. IT is not playing around here. You can't say, "Maybe we will do SOA, if we can figure it out, or maybe we won’t. We’ll just do things the old way, where we are siloed and we keep on going."

Instead of just running around and not being able to use technology, we want to have a governance plan in place, saying “This is how we’re dealing with problems. Here is how the technology will rise to these challenges." And, we make it a matter of policy. So now, instead of just having to wing it when you have some sort of issue, there is an infrastructure in place for helping you deal with issues as they come about.

A key to this is the management challenge. As management technology improves, it is less and less about just monitoring stuff, and more and more about being able to deal with issues as a matter of policy, where your policy is in place for dealing with problems that you can’t predict -- and those are the most challenging ones. That’s what we see happening over time.

What’s happening in the management standards world is a pissing match between the big vendors. You have the Java guys wanting to this and then Microsoft guys wanting to do that, and nobody listens to them, because they can't agree with each other. So, they'll realize, "Hey, all of our customers are ignoring us. We'd better get our act together." It’s become this big political thing that’s just slowing whole thing down.

From the enterprise perspective, you don’t have to wait around for the vendors to grow up. You can get stuff done today. This isn’t going to stop you from being successful with SOA initiatives today. It might mean that two products you buy off the shelf might not interoperate as well you like. That has to be part of your plan. It might mean you have to come up with your own internal standards for the time being.

As companies move closer to SOA, it forces them to grow up. It forces them to think across boundaries. In the past, complexity has forced companies to divvy up issues into small compartments, put a box around them, and assign people to that item of complexity.

But that has stifled the ability of interoperability and of addressing things holistically, of being fleet and agile. It’s made them brittle, has made them slower, and has made things expensive. SOA forces companies to start binding what happens in pre-production to post-production, what happens in an application with what happens in an infrastructure, and what happens on a service level from an outside provider to what happens as a shared service internally.

There are great risks if you try to do SOA without growing up, but there are super opportunities if it’s done properly. It can elevate IT as a function within the organization from being an inhibitor to the absolute enablement for new business and growth and opportunity.

Listen to the podcast. Read a full transcript of the discussion. Sponsor: Tidal Software.

Microsoft opens kimono to show athletic supporter with iron-clad cup

The "if you can't beat 'em, join 'em" mood at Microsoft has gotten more of an open source flavor. Soon, developers (and other interested parties) will see more source code under the .NET and Visual Studio 2008 covers. This is interesting.

But by pulling back the kimono, Microsoft has only shown an iron-clad cup over the family jewels. Look but don't touch. No chance of legally changing the code, or redistributing it, only learn and learn well. Sorta open source.

It's a pretty big deal, I guess. I shows that even Microsoft recognizes that certain elements of the open source credo make sense, if not enough cents. The mantra of open source is killing software and has no role in the real world -- well, I doubt we'll seeing much more of that.

It's like George Bush saying he's no longer against raising taxes, but has no intention of actually doing it. It's the thought that counts!

Is this a slippery slope beyond the FUD factor, however? Will such code exposure lead to outright dancing in the moonlight someday? How many products will they give the peak-a-boo treatment too? When will such openness become a security ... err, legal, risk?

Well hackers and competitors already get a lot of gropes at the code, anyway. This just gives the honest people a thrill. Or maybe there's more to the gesture. Could this be a set-up that anyone who looks at the code and comes up with way to skirt using the Windows runtime when they enjoy the splendors of the development environment?

I expect that with such access to the goodies that more folks will want to develop around the framework and tools. And while that inevitably leads to more sales of the runtime, the decoupling of the pre-production and the post-production continues. This can only hasten the trend.

One has to wonder how Microsoft's lawyers will interpret the code "advances" over time.

IBM 'continuum' helps companies crawl-walk-run along the SOA path

IBM today unleashed a barrage of announcements that cover services and software to enable a crawl-walk-run strategy for enterprises, as they move into the services-oriented architecture (SOA) world.

The new offerings range from a "SOA Sandbox," which will allow developers to "play" with SOA before using it in a production environment, to WebSphere enhancements, and finally, to an unveiling of the new IBM Optim, a data governance application from recently acquired Princeton Softech.

Key to the crawl-walk-run approach is what IBM calls their SOA Continuum, which takes companies from the very basic -- involving only focused, high-ROI projects -- to the most advanced, in which technology becomes invisible and more than 80 percent of business functions are delivered as services -- and more than 50 percent of those are re-used.

I sort of remember the logic from an earlier blog. Glad they agree.

Recognizing the need for what they've dubbed the Globally Integrated Enterprise, IBM has also introduced assessment tools, including a Benchmark Wizard, which is based on key agility indicators and is loaded with 1,100 qualitative business indicators and uses best practices derived from 1,600 case studies. This will allow an enterprise to determine how it stacks up against the industry in general.

The new offerings also address several key stumbling blocks along the path to SOA-based agility. One such sticking point is how systems respond when services from various sources may be unavailable. IBM says its updates WebSphere Process Server has extensive compensation support, allowing process to recover reliably when target applications are unavailable.

Another key concern is exposing sensitive or personal data when services access databases. The newly acquired Optim product is designed to identify and protect private data in complex application environments.

IBM also announced enhancement for process integrity to several products, including:

• Message Broker and MQ
• Tivoli Composite Application Manager for SOA
• WebSphere DataPower XML Security Gateway
• IBM Information Server

IBM is taking an "all things to all people" approach and offering new SOA configurations, designed to help enterprises use legacy and packaged applications in a SOA environment. These include best practices and step-by-step implementation guides.

To help developers get started -- before even starting to crawl along the SOA path -- IBM has set up a SOA Sandbox, a free test bed on the developerWorks site. The sandbox, where potential users can download trial software or can "play" online in a hosted environment, includes software, tutorials, quick-start guides, and best-practices guidance.

As another aid to companies, IBM is offering "SOA Healthchecks," workshops to assess and diagnose applications, services and infrastructure to determine application reuse, security, and infrastructure flexibility.

Look for this opening salvo to increase in crescendo over the next few months, as IBM rolls out these and more SOA offerings at 500 events, including a customer Webcast on Oct. 9, and a series of announcements to create a drumbeat about the latest SOA end-to-end initiative.

Engagement on SOA has to take place in countless ways. IBM seems to be up to the task, or least is ready to try.

Adobe pushes foward on enabling RIA ecosystem with Flex 3, Adobe AIR betas

The world of rich Internet applications (RIA) got a little richer on Monday when Adobe Systems released the beta versions of Flex 3 and the Flex SDK with enhanced features for evelopers.

The beta versions are in place of the alpha release, which was scheduled for this month, but which has been pushed out to the first quarter of 2008 to coincide with the release of Adobe AIR 1.0. The AIR beta is also available.

Enhanced capabilities of the Flex beta include support for ASP.NET, including new data wizards that allow developers to look at data tables and create a new Flex application from a SQL database. Developers who consume Web services can view the WSDL files and automatically generate code for invoking operations.

Adobe AIR now supports background applications and system tray notifications, allowing the app to run in the background. Among its other features are:

• Synchronous APIs for embedded local database
• Greater control of windows and menus
• Content protection for video, HTML improvements
• Application and runtime enhancements such as improved install process and automatic updates for the runtime.

Check out Matt Chotin's blog at Adobe for a complete rundown on the latest enhancements and updates.

Also, keep an eye out for Adobe to announce a new Adobe Developer Connection, where the Adobe community can get newsletters, form a professional network, participate in forums, and get special offers.

The Adobe AIR and the Flex betas can be downloaded from the Adobe Labs site, which also has a pre-release version of Adobe Media Player, which will be free to end users and will allow companies to distribute Web 2.0 content in video format. Major television broadcasters and other content producers have already agreed to support the Adobe format.

Adobe made the announcements at Adobe Max 2007. Among other news from the event:

Adobe has signed a definitive agreement to acquire Virtual Ubiquity and its online word processor, Buzzword, built with Adobe Flex and leveraging Flash Player. It's soon to be available on Adobe AIR. Buzzword has integral collaboration capabilities that allows multiple authors to edit and comment on documents from anywhere at anytime. Because it will run on Adobe AIR, the application will offer users a hybrid online/offline experience for working with documents.

Adobe and Business Objects have agreed to jointly undertake multiple initiatives to drive product interoperability and optimization, technology adoption, and product distribution.

One of the key initiatives is the development of a Business Objects Xcelsius Connector to Adobe LiveCycle Data Services ES.

Adobe is posing a strong alternative to other platform approaches to RIA and Enterprise 2.0 interfaces, most notably against Microsoft. It's just one of the thousands of daggers pointed at Redmond nowadays, but its cuts could go deep.

I'd like to see Adobe make more partnerships with those vendors supporting SOA activities. More ease and integration of RIAs and SOA infrastructure could be powerful. Microsoft obviously thinks so. I wonder why IBM and Adobe are not closer, at least in their more blatant go-to-marker campaigns.

I'd also like to see Adobe move more quickly on the completeness of the platform approach. These RIAs are catching on fast.

Software AG consolidates governance products for SOA

Software AG took the "big picture" view of governance for service-oriented architecture (SOA) with the announcement today of the CentraSite Governance Edition.

The approach aims to enable governance more broadly across the enterprise, with policy enforcement capabilities building on the webMethods Infravio X-Registry (acquired by SAG with Infravio earlier this year). Enforcement features are built directly into the platform, which offers a richer and more extensible metadata repository, a more intuitive Web2.0 interface, and wizard-driven templates.

While SOAs can help companies achieve much-needed business agility -- by flexibly arranging and repurposing existing and new IT assets, both from within and outside the enterprise -- that agility comes at the price of greater complexity.

One stumbling block has been the governance of SOA assets and processes, along with the ability to set, adjust and enforce policies. SOAs will require finer management of assets as they access and expose ever more enterprise data and interactions.

Encompassing both a UDDI (Universal Description, Discovery and Integration protocol), v. 3.0 compliant registry and a fully extensible, JAXR complaint (Java API for XML Registries) repository for maintaining associated metadata and policies, CentraSite Governance Edition can be implemented as the design-time, run-time and change-time governance platform for any heterogeneous environment.

It also acts as a policy hub for run-time enforcement with an integrated run-time policy enforcement point, webMethods X-Broker, and standards-based support for additional third-party applications used to mediate transactions betweens service providers and consumers.

A new feature in CentraSite Governance Edition, available now, is Active Policy, which automates SOA processes and simplifies end-user adoption. It is also pre-loaded with more than 80 pre-defined best practices that help streamline end-user adoption.

Other features include:

• Unified lifecycle governance
• Enhanced repository
• Open metadata model
• Enhanced change-time governance
• Customized views
• SOA federation
• Standards support
  

Tony Baer at CBR Online sees Software AGs move along its webMethods roadmap as being "right paced" and even takes a look into the future:

"Software AG is moving very deliberately to execute on the roadmap for converging products from the acquired webMethods. With announcements right paced at about every four weeks, this is the second such announcement to come out of Software AG since it unveiled the roadmap back in August.

"The company is also being upfront, in that, while it is making the products look cosmetically the same at the UI level and in branding, it admits that the real convergence will happen next year. At this point, both have, or will have, common Ajax-based rich user interfaces. Both rely also on the JAR (Java API for XML Registries), so access in also via a common method."

Integration infrastructure approaches adjust to new world of SaaS and shared services

Read a full transcript of the discussion. Listen to the podcast. Sponsor: Cape Clear Software.

Change is afoot for the role and requirements of integration for modern software-as-a-service (SaaS) providers and enterprises adopting shared services models. Reuse is becoming an important issue, as are patterns of automation.

The notion of reuse of integration -- with added emphasis on integration as a service -- has prompted a different approach to integration infrastructure. The new demand is driven by ecologies of services, some from the Web "cloud," as well as the need to efficiently scale the delivery of services and applications composed of many disparate component services.

Integrations require reusable patterns, high performance, as well as many different means of access from clients. As a result Cape Clear Software has this week unveiled a new major version of its enterprise service bus (ESB), Cape Clear 7.5, with an emphasis on:

• A new graphical editor, the SOA Assembly Editor, an Eclipse-based tool to graphically clip together elements of integrations.

• Multi-tenanting additions to the ESB that allow segmentation of integrations, data, and reporting, as well as segmenting use and reuse of integrations on reporting and management of integrations based on the identities of inbound customers, clients, or businesses.

• A Business Process Execution Language (BPEL) management system with tools to monitor transactions, and repair transactions when they fail, to allow for rebuilding previous business information and ensure transactional integrity in running and maintaining large enterprise-class BPEL deployments.
  

To help better understand the new landscape for integration models, I recently moderated a sponsored podcast discussion with Phil Wainewright, an independent consultant, director of Procullux Ventures, and fellow ZDNet SaaS blogger, as well as Annrai O’Toole, CEO of Cape Clear Software.

Here are some excerpts:

... We're getting more sophisticated about SaaS, because it's being taken on board in a whole range of areas within the enterprise, and people want to do integration.

There are two forms of integration coming to the fore. The first is where data needs to be exchanged with legacy applications within the enterprise. The second form of integration that we see -- not at the moment, but it’s increasingly going to be an issue -- is where people want to integrate between different services coming in from the cloud. It’s a topic that’s familiar when we talk about mashups, fairly simple integrations of services that are done at the browser level. In the enterprise space, people tend to talk about composite applications, and it seems to be more difficult when you are dealing with a range of data sources that have to be combined.

People have realized that if you're doing integration to each separate service that's out there, then you're creating the same point-to-point spaghetti that people were trying to get away from by moving to this new IT paradigm. People are starting to think that there's a better way of doing this. If there's a better way of delivering the software, then there ought to be a better way of integrating it together as well.

Therefore, they realize that if we share the integration, rather than building it from scratch each time, we can bring into the integration field some of the benefits that we see with the shared-services architecture or SaaS. ... The new generation of SaaS providers, are really talking about a shared infrastructure, where the application is configured and tailored to the needs of individual customers. In a way, they’re segmented off from the way the infrastructure works underneath.

When you build an integration, you always end up having to customize it in some way for different customers. Customers will have different data formats. They’ll want to access it slightly differently. Some people will want to talk to it over SOAP. Some won't, and they’ll want to use something like REST. Or they might be going backwards and are only able to send it FTP drops, or something like that.

Multi-tenanting is one solution to the problem. The other is what we call multi-channel, which is the ability to have an integration, and make it available with different security policies, different transports, and different transformations going in and out.

A combination of multi-tenanting and multi-channeling allows you to build integrations once, make them accessible to different users, and make them accessible in different ways for each of those different customers. It gives you the scalability and reuse you need to make this model viable.

One point worth bearing in mind here is that this problem is going to get solved, because the economic reality of it suggests that we must solve this. One, the payoff for getting it right is huge. Second, the whole model of SaaS won’t be successful, unless we skin the integration problem. We don’t want the world to be limited to just having Salesforce.com with its siloed application.

We want SaaS to be the generic solution for everybody. That’s the way the industry is going, and that can only happen by solving this problem. So, we’re having a good stab at it, and I'll just briefly address some of the things that I think enable us to do it now, as opposed to in the past. First, there is a standardization that’s taken place. A set of standards has been created around SOA, giving us the interoperability platform that makes it possible in a way that was never possible before. Second is an acceptance of this shared-services, hosted model.

Years ago, people would have laughed at you and said, "I’m going to trust all my customer data to a provider in the cloud?" But, they’re doing it happily because of the economics of it. The whole trend toward trusting people with outsourced offerings means that the people will be more likely to trust integrations out there, because a lot of the technology to do this has been around for quite some time.

In enterprises you’re seeing this big move to virtualization and shared services. They’re saying, "Why are we having development teams build integration in all these branch offices at all these locations around the world? It’s extremely wasteful. It's a lot of skill that we've got to push out, and there are a lot of things that go wrong with these. Can't we consolidate all of those into a centralized data center? We’ll host those integrations for those individual business units or those at departments, but we'll do it here. We’ve got all the expertise in one place."

Those guys are delighted, because at the individual local level they don’t maintain all the costs and all the complexity of dealing with all the issues. It’s hosted out in their internal cloud. We haven't seen enough data points on that, but this hosted integration model can work. We’ve got it working for pure entities in SaaS companies like Workday, and we’ve got it working for a number of large enterprises. There is enough evidence for us to believe that this is really going to be the way forward for everybody in the industry.

Read a full transcript of the discussion. Listen to the podcast. Sponsor: Cape Clear Software.

Integrien deepens analytics, betters interoperability and userability in Q4's Alive 6.0 release

Read a full transcript of the discussion. Listen to the podcast. Sponsor: Integrien Corp.

The movement of IT and systems management to the end-to-end business service value level has been a long time in coming. Yet the need has never been higher. Enterprises and on-demand application providers alike need to predict how systems will behave under a variety of conditions.

Rather than losing control to ever-increasing complexity -- and gaining less and less insight into the root causes of problematic applications and services -- operators must gain the ability to predict and prevent threats to the performance of their applications and services. Firefighting against applications performance degradation in a dynamic service-oriented architecture (SOA) just won't cut it.

By adding real-time analytics to their systems management practices, IT operators can determine the normal state of how systems should be performing. Then, by measuring the characteristics of systems under many conditions over time, administrators can gain predictive insights into their entire operations, based on a business services-level of performance and demand. They can stay ahead of complexity, and therefore contain the costs of ongoing high-performance applications delivery.

I recently had a podcast discussion with Mazda Marvasti, the CTO of Integrien Corp., on managing complexity by leveraging probabilistic systems management and remediation. I learned that Integrien's Alive suite uses probabilistic analysis to predict IT systems problems before costly applications outages. Furthermore, I received some details on the next Alive 6.0 release in Q4 of this year.

Here are some excerpts:

Can you give us some sense of the direction that the major new offerings within the Alive product set will take?

Basically, we have three pillars that the product is based on. First is usability. That's a particular pet peeve of mine. I didn't find any of the applications out there very usable. We have spent a lot of time working with customers and working with different operations groups. ... The second piece is interoperability. The majority of the organizations that we go to already have a whole bunch of systems, whether it be data collection systems, event management systems, or configuration management databases, etc.

Our product absolutely needs to leverage those investments -- and they are leveragable. But even those investments in their silos don’t produce as much benefit to the customer as a product like ours going in there and utilizing all of that data that they have in there, and bringing out the information that’s locked within it.

The third piece is analytics. What we have in the product coming out is scalability to 100,000 servers. We've kind of gone wild on the scalability side, because we are designing for the future. Nobody that I know of right now has that kind of a scale, except maybe Google, but theirs' is basically the same thing replicated thousands of times over, which is different than the enterprises we deal with, like banks or health-care organizations.

A single four-processor Xeon box, with Alive installed on it, can run real-time analytics for up to 100,000 devices. That’s the level of scale we're talking about. In terms of analytics, we've got three new pieces coming out, and basically every event we send out is a predictive event. It’s going to tell you this event occurred, and then this other set of events have a certain probability within a certain timeframe to occur.

Not only that, but then we can match it to what we call our "finger printing." Our finger printing is a pattern-matching technology that allows us to look at patterns of events and formulate a particular problem. It indicates particular problems and those become the predictive alerts to other problems.

Now, with SOA and virtualization moving into application-development and data-center automation, there is a tremendous amount of complexity in the operations arena. You can’t have the people who used to have the "tribal knowledge" in their head determining where the problems are coming from or what the issues are.

The problems and the complexity have gone beyond the capability of people just sitting there in front of screens of data, trying to make sense out of it. So, as we gained efficiency from application development, we need consistency of performance and availability, but all of this added to the complexity of managing the data center.

That’s how the evolution of the data center went from being totally deterministic, meaning that you knew every variable, could measure it, and had very specific rules telling you if certain things happened, and what they were and what they meant -- all the way to a non-deterministic era, which we are in right now.

Now, you can't possibly know all the variables, and the rules that you come up with today may be invalid tomorrow, all just because of change that has gone on in your environment. So, you cannot use the same techniques that you used 10 or 15 years ago to manage your operations today. Yet that’s what the current tools are doing. They are just more of the same, and that’s not meeting the requirements of the operations center anymore.

I’ve been working on these types of problems for the past 18 years. Since graduate school, I’ve been analyzing data extraction of information from disparate data. I went to work for Ford and General Motors -- really large environments. Back then, it was client-servers and how those environments were being managed. I could see the impending complexity, because I saw the level of pressure that there was on application developers to develop more reusable code and to develop faster with higher quality.

The run book is missing that information. The run book only has the information on how to clean it up after an accident happens.

That’s the missing piece in the operations arena. Part of the challenge for our company is getting the operations folks to start thinking in a different fashion. You can do it a little at a time. It doesn’t have to be a complete shift in one fell swoop, but it does require that change in mentality. Now that I am actually forewarned about something, how do I prevent it, as opposed to cleaning up after it happens.

Read a full transcript of the discussion. Listen to the podcast. Sponsor: Integrien Corp.