Part 3 of 4: Web data services--Here's why text-based content access and management plays crucial role in real-time BI

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. View a full transcript or download a copy. Learn more. Sponsor: Kapow Technologies.

Text-based content and information from across the Web are growing in importance to businesses. The need to analyze web-based text in real-time is rising to where structured data was in importance just several years ago.

Indeed, for businesses looking to do even more commerce and community building across the Web, text access and analytics forms a new mother lode of valuable insights to mine.

As the recession forces the need to identify and evaluate new revenue sources, businesses need to capture such web data services for their business intelligence (BI) to work better, deeper, and faster.

In this podcast discussion, Part 3 of a series on web data services for BI, we discuss how an ecology of providers and a variety of content and data types come together in several use-case scenarios.

In Part 1 of our series we discussed how external data has grown in both volume and importance across the Internet, social networks, portals, and applications. In Part 2, we dug even deeper into how to make the most of web data services for BI, along with the need to share those web data services inferences quickly and easily.

Our panel now looks specifically at how near real-time text analytics fills out a framework of web data services that can form a whole greater than the sum of the parts, and this brings about a whole new generation of BI benefits and payoffs.

To help explain the benefits of text analytics and their context in web data services, we're joined by Seth Grimes, principal consultant at Alta Plana Corp., and Stefan Andreasen, co-founder and chief technology officer at Kapow Technologies. The discussion is moderated by me, Dana Gardner, principal analyst at Interarbor Solutions.

Here are some excerpts:

Grimes: "Noise free" is an interesting and difficult concept when you're dealing with text, because text is just a form of human communication. Whether it's written materials, or spoken materials that have been transcribed into text, human communications are incredibly chaotic ... and they are full of "noise." So really getting to something that's noise-free is very ambitious.

... It's become an imperative to try to deal with the great volume of text -- the fire hose, as you said -- of information that's coming out. And, it's coming out in many, many different languages, not just in English, but in other languages. It's coming out 24 hours a day, 7 days a week -- not only when your business analysts are working during your business day. People are posting stuff on the web at all hours. They are sending email at all hours.

If you want to keep up, if you want to do what business analysts have been referring to as a 360-degree analysis of information, you've got to have automated technologies to do it.

... There are hundreds of millions of people worldwide who are on the Internet, using email, and so on. There are probably even more people who are using cell phones, text messaging, and other forms of communication.

If you want to keep up, if you want to do what business analysts have been referring to as a 360-degree analysis of information, you've got to have automated technologies to do it. You simply can't cope with the flood of information without them.

Fortunately, the software is now up to the job in the text analytics world. It's up to the job of making sense of the huge flood of information from all kinds of diverse sources, high volume, 24 hours a day. We're in a good place nowadays to try to make something of it with these technologies.

Andreasen: ... There is also a huge amount of what I call "deep web," very valuable information that you have to get to in some other way. That's where we come in and allow you to build robots that can go to the deep web and extract information.

... Eliminating noise is getting rid of all this stuff around the article that is really irrelevant, so you get better results.

The other thing around noise-free is the structure. ... The key here is to get noise-free data and to get full data. It's not only to go to the deep web, but also get access to the data in a noise-free way, and in at least a semi-structured way, so that you can do better text analysis, because text analysis is extremely dependent on the quality of data.

Grimes: ... [There are] many different use-cases for text analytics. This is not only on the Web, but within the enterprise as well, and crossing the boundary between the Web and the inside of the enterprise.

Those use-cases can be the early warning of a Swine flu epidemic or other medical issues. You can be sure that there is text analytics going on with Twitter and other instant messaging streams and forums to try to detect what's going on.

... You also have brand and reputation management. If someone has started posting something very negative about your company or your products, then you want to detect that really quickly. You want early warning, so that you can react to it really quickly.

We have some great challenges out there, but . . . we have great technologies to respond to those challenges.

We have a great use case in the intelligence world. That's one of the earliest adopters of text analytics technology. The idea is that if you are going to do something to prevent a terrorist attack, you need to detect and respond to the signals that are out there, that something is pending really quickly, and you have to have a high degree of certainty that you're looking at the right thing and that you're going to react appropriately.

... Text analytics actually predate BI. The basic approaches to analyzing textual sources were defined in the late '50s. Actually, there is a paper from an IBM researcher from 1958, that defines BI as the analysis of textual sources.

...[Now] we want to take a subset of all of the information that's out there in the so-called digital universe and bring in only what's relevant to our business problems at hand. Having the infrastructure in place to do that is a very important aspect here.

Once we have that information in hand, we want to analyze it. We want to do what's called information extraction, entity extraction. We want to identify the names of people, geographical location, companies, products, and so on. We want to look for pattern-based entities like dates, telephone numbers, addresses. And, we want to be able to extract that information from the textual sources.

Suitable technologies

All of this sounds very scientific and perhaps abstruse -- and it is. But, the good message here is one that I have said already. There are now very good technologies that are suitable for use by business analysts, by people who aren't wearing those white lab coats and all of that kind of stuff. The technologies that are available now focus on usability by people who have business problems to solve and who are not going to spend the time learning the complexities of the algorithms that underlie them.

Andreasen: ... Any BI or any text analysis is no better than the data source behind it. There are four extremely important parameters for the data sources. One is that you have the right data sources.

There are so many examples of people making these kind of BI applications, text analytics applications, while settling for second-tier data sources, because they are the only ones they have. This is one area where Kapow Technologies comes in. We help you get exactly the right data sources you want.

The other thing that's very important is that you have a full picture of the data. So, if you have data sources that are relevant from all kinds of verticals, all kinds of media, and so on, you really have to be sure you have a full coverage of data sources. Getting a full coverage of data sources is another thing that we help with.

Noise-free data

We already talked about the importance of noise-free data to ensure that when you extract data from your data source, you get rid of the advertisements and you try to get the major information in there, because it's very valuable in your text analysis.

Of course, the last thing is the timeliness of the data. We all know that people who do stock research get real-time quotes. They get it for a reason, because the newer the quotes are, the surer they can look into the crystal ball and make predictions about the future in a few seconds.

The world is really changing around us. Companies need to look into the crystal ball in the nearer and nearer future. If you are predicting what happens in two years, that doesn't really matter. You need to know what's happening tomorrow.

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. View a full transcript or download a copy. Learn more. Sponsor: Kapow Technologies.

Role of governance plumbed in Nov. 10 webinar on managing hybrid and cloud computing types

I'll be joining John Favazza, vice president of research and development at WebLayers, on Nov. 10 for a webinar on the critical role of governance in managing hybrid cloud computing environments.

The free, live webinar begins at 2 p.m. EDT. Register at https://www2.gotomeeting.com/register/695643130. [Disclosure: WebLayers is a sponsor of BriefingsDirect podcasts.]

Titled "How Governance Gets You More Mileage from Your Hybrid Computing Environment,” the webinar targets enterprise IT managers, architects and developers interested in governance for infrastructures that include hybrids of cloud computing, software as a service (saaS) and service-oriented architectures (SOA). There will be plenty of opportunity to ask questions and join the discussion.

Organizations are looking for more consistency across IT-enabled enterprise activities, and are finding competitive differentiation in being able to best manage their processes more effectively. That benefit, however, requires the ability to govern across different types of systems and infrastructure and applications delivery models. Enforcing policies, and implementing comprehensive governance, acts to enhance business modeling, additional services orientation, process refinement, and general business innovation.

Increasingly, governance of hybrid computing environments establishes the ground rules under which business activities and processes -- supported by multiple and increasingly diverse infrastructure models -- operate.

Developing and maintaining governance also fosters collaboration between architects, those building processes and solutions for companies, and those operating the infrastructure -- be it supported within the enterprise or outside. It also sets up multi-party business processes, across company boundaries, with coordinated partners.

Cambridge, Mass.-based WebLayers provides a design-time governance platform that helps centralize policy management across multiple IT domains -- from SOA through mainframe and cloud implementations. Such governance clearly works to reduce the costs of managing and scaling such environments, individually and in combination.

In the webinar we'll look at how structured policies, including extensions across industry standards, speeds governance implementations and enforcement -- from design-time through ongoing deployment and growth.

So join me and Favazza and me at 2 p.m. ET on Nov. 10 by registering at https://www2.gotomeeting.com/register/695643130.

HP takes converged infrastructure a notch higher with new data warehouse appliance

Hewlett-Packard (HP) on Wednesday announced new products, solutions and services that leaves the technology packaging to them, so users don't have to.

HP Neoview Advantage, HP Converged Infrastructure Architecture, and HP Converged Infrastructure Consulting Services are designed to help organizations drive business and technology innovations at lower total cost via lower total hassle. [Disclosure: HP is a sponsor of BriefingsDirect podcasts.]

HP’s measured focus

HP isn’t just betting on a market whim. Recent market research it supported reveals that more than 90 percent of senior business decision makers believe business cycles will continue to be unpredictable for the next few years — and 80 percent recognize they need to be far more flexible in how they leverage technology for business.

The same old IT song and dance doesn't seem to be what these businesses are seeking. Nearly 85 percent of those surveyed cited innovation as critical to success, and 71 percent said they would sanction more technology investments -- if they could see how those investments met their organization’s time-to-market and business opportunity needs.

Cost nowadays is about a lot more than the rack and license. The fuller picture of labor, customization, integration, shared services suppport, data-use-tweaking and inevitable unforeseen gotchas need to be better managed in unison -- if that desired agility can also be afforded (and sanctioned by the bean-counters).

HP said its new offerings deliver three key advantages:

• Improved competitiveness and risk mitigation through business data management, information governance, and business analytics
  
  
• Faster time to revenue for new goods and services
  
  
• The ability to return to peak form, after being compressed or stretched.

The Neoview advantage

First up, HP Neoview Advantage, the new release of the HP Neoview enterprise data warehouse platform, which aims to help organizations respond to business events more quickly by supporting real-time insight and decision-making.

HP calls the performance, capacity, footprint and manageability improvements dramatic and says the software also reduces the total cost of ownership (TCO) associated with industry-standard components and pre-built, pre-tested configurations optimized for warehousing.

HP Neoview Advantage and last year's Exadata product (produced in partnership with Oracle) seem to be aimed at different segments. Currently, HP Neoview Advantage is a "very high end database," whereas Exadata is designed for "medium to large enterprises," and does not scale to the Neoview level, said Deb Nelson, senior vice president, Marketing, HP Enterprise Business.

A converged infrastructure

Next up, HP Converged Infrastructure architecture. As HP describes it, the architecture adjusts to meet changing business needs, specifically what HP calls “IT sprawl,” which it points to as the key culprit in raising technology costs for maintenance that could otherwise be used for innovation.

HP touts key benefits of this new architecture. First, the ability to deploy application environments on the fly through shared service management, followed closely by lower network costs and less complexity. The new architecture is optimized through virtual resource pools and also improves energy integration and effectiveness across the data center by tapping into data center smart grid technology.

Finally, HP is offering Converged Infrastructure Consulting Services that aim to help customers transition from isolated product-centric technologies to a more flexible converged infrastructure. The new services leverage HP’s experience in shared services, cloud computing, and data center transformation projects to let customers design, test and implement scalable infrastructures.

Overall, typical savings of 30 percent in total costs can be achieved by implementing Data Center Smart Grid technologies and solutions, said HP.

With these moves to converged infrastructure, HP is filling out where others are newly treading. Cisco and EMC this week announced packaging partnerships that seek to deliver simiar convergence benefits to the market.

"It's about experience, not an experiment," said Nelson.

BriefingsDirect contributor Jennifer LeClaire provided editorial assistance and research on this post.

Aster Data architects application logic with data for speeded-up analytics processing en masse

In real estate, the mantra is "location, location, location." The same could be said for the juxtaposition of applications logic and data. With enterprise data growing at an explosive rate, having applications separate from the mountains of data that they rely on has resulted in massive data movement -- increasing latency and restricting due analysis.

Aster Data, which provides massively parallel processing (MPP) data management, has tackled the location problem head-on with the announcement this week of Aster Data Version 4.0, (along with Aster nCluster System 4.0), a massively parallel application-data server that allows companies to embed applications inside an MPP data warehouse. This is designed to speed the processing of terabytes to petabytes of data.

The latest offering from the San Carlos, Calif., company fully parallelizes both data and a wide variety of analytics applications in one system. This provides faster analysis for such data-heavy applications as real-time fraud detection, customer behavior modeling, merchandising optimization, affinity marketing, trending and simulations, trading surveillance, and customer calling patterns.

While both data and applications reside in the same system, they are independent of one another, but both execute as "first-class citizens" with their respective data and application management services.

Resource sharing

The Aster Data Application Server is responsible for managing and coordinating activities and resource sharing in the cluster. It also acts as a host for the application processing and data inside the cluster. In its role as data host, it manages incremental scaling, fault tolerance and heterogeneous hardware for application processing.

Aster Data Version 4.0 provides application portability, which allows companies to take their existing Java, C, C++, C#, .NET, Perl and Python applications, MapReduce-enable them and push them down into the data.

The Dynamic Workload Management (WLM) helps support hundreds of concurrent mixed workloads that can span interactive and batch data queries, as well as application execution. Includes granular rule-based prioritization of workloads and dynamic allocation and re-allocation of resources.

Other features include:

• Trickle feeds for granular data loading and interactive queries with millisecond response times
  
  
• New online partition splitting capabilities to allow infinite cost-effective scaling
  
  
• Dual-stage query optimizer, which ensures peak performance across hundreds to thousands of CPU cores
  
  
• Integrations with leading business intelligence (BI) tools and Hadoop.

More companies want to bring more data to bear on more BI problems. While Aster's benefits and value may be used for high-end and esoteric analytics uses now, I fully expect that there data-intense architectures will be finding more uses. The price, too, is dropping, making the use of such systems more affordable.

Many of the core users of high-end analytics are also moving on architecture-wise. The systems designed five or more years ago will not meet the needs of five or even a few years from now.

What's really cool about Aster Data's approach is the analytics apps can be used, and the languages and query semantics most familiar to users can be used with the new systems and architectures.

I suppose we should also expect more of these analytics engines to become available as services, aka cloud services. That would allow joins of more data sets and they the massive analytics applications can open up even more BI cans of worms.

Survey: Virtualization and physical infrastructures need to be managed in tandem

If your company uses test and development infrastructures as a proving ground for shared services, virtualization and private cloud environments, you’re not alone. More companies are moving in that direction, according to a Taneja Group survey.

Yet underlying the use of the newer infrastructure approaches lies a budding challenge. The recent Taneja Group survey of senior IT managers working on test/dev infrastructures at North American firms found that 72 percent of respondents said virtualization on its own doesn’t address their most important test/dev infrastructure challenges. Some 55 percent rate managing both virtual and physical resources as having a high or medium impact on their success. The market is clearly looking for ways to bridge this gap.

Sharing physical and virtual infrastructures

Despite the confusion in the market about the economics of the various flavors of cloud computing, Dave Bartoletti, a senior analyst and consultant at Taneja Group, says one thing is clear: Enterprises are comfortable with, and actively sharing, both physical and virtual infrastructures internally.

“This survey reaffirms that shared infrastructure is common in test/dev environments and also reveals it’s increasingly being deployed for production workloads,” Bartoletti says. "Virtualization is seen as a key enabling technology. But on its own it does not address the most important operational and management challenges in a shared infrastructure.”

Half the survey respondents are funding projects starts in 2009. Another 66 percent of respondents will have funded a project started by the end of 2010.

Noteworthy is the fact that 92 percent of test/dev operations are using shared infrastructures, and companies are making significant investments in infrastructure-sharing initiatives to address the operational and budgetary challenges. Half the survey respondents are funding projects in 2009. Another 66 percent of respondents will have funded a project started by the end of 2010.

The survey reveals most firms are turning to private cloud infrastructures to support test/dev projects, and that shared infrastructures are beginning to bridge the gap between pre-production and production silos. A full 30 percent are sharing resource pools between both test/dev and production applications. This indicates a rising comfort level with sharing infrastructure within IT departments.

Virtualization’s cost and control issues

Although 89 percent of respondents use virtualization for test/dev, more than half have virtualized less than 25 percent of their servers. That’s because virtualization adds several layers of control and cost issues that need to be addressed by sharing, process, workflow and other management capabilities in order to fully maximize and integrate both virtual and physical infrastructures.

“Test/Dev environments are one of the most logical places for organizations to begin implementing private clouds and prove the benefits of a more elastic, self service, pay-per-use service delivery model,” says Martin Harris, director Product Management at Platform Computing. “We’ve certainly seen this trend among our own customers and have found that additional management tools enabling private clouds are required to effectively improve business service levels and address cost cutting initiatives.” [Disclosure: Platform Computing is a sponsor of BriefingsDirect podcasts.]

Despite the heavy internal investments, however, 82 percent of respondents are not using hosted environments outside their own firewalls. The top barriers to adoption: Lack of control and immature technology.

BriefingsDirect contributor Jennifer LeClaire provided editorial assistance and research on this post.

You'll be far better off in a future without enterprise software

This guest post comes courtesy of Ronald Schmelzer, senior analyst at ZapThink.

By Ronald Schmelzer

The conversation about the role and future of enterprise software is a continuous undercurrent in the service oriented architecture (SOA) conversation. Indeed, ZapThink’s been talking about the future of enterprise software in one way or another for years.

So, why bother bringing up this topic again, at this juncture? Has anything changed in the marketplace? Can we learn something new about where enterprise software is heading? The answer is decidedly "yes" to the latter two questions. And this might be the right time to seriously consider acting on the very things we’ve been talking about for a while.

The first major factor is significant consolidation in the marketplace for enterprise software. While a decade or so ago there were a few dozen large and established providers of different sorts of enterprise software packages, there are now just a handful of large providers, with a smattering more for industry-specific niches.

We can thank aggressive M&A activity combined with downward IT spending pressure for this reality. As a result of this consolidation, many large enterprise software packages (such as enterprise resource planning (ERP), customer relationship management (CRM), supply chain management (SCM) offerings) have been eliminated, are in the process of being phased out, or are getting merged (or “fused”) with other solutions.

Many companies rationalized the spending of millions of dollars on enterprise software applications because the costs could be amortized over a decade or more of usage, and they could claim that these enterprise software applications would be cheaper, in the long run, than building and managing their existing custom code. But, we’ve now had a long enough track record to realize that the result of mass consolidation, need for continuous spending, and inflexibility is causing many companies to reconsider that rationalization.

We can thank expensive, cumbersome, and tightly-coupled customization, integration, and development for this lack of innovation in enterprise software.

Furthermore, by virtue of their weight, significance in the enterprise environment, and astounding complexity, enterprise software solutions are much slower to adopt and adapt to new technologies that continuously change the face of IT.

We refer to this as the “enterprise digital divide.” You get one IT user experience when you are at home and use the Web, personal computing, and mobile devices and applications and a profoundly worse experience when you are at work. It’s as if the applications you use at work are a full decade behind the innovations that are now commonplace in the consumer environment. We can thank expensive, cumbersome, and tightly coupled customization, integration, and development for this lack of innovation in enterprise software.

In addition, no company can purchase and implement an enterprise software solution “out of the box.” Not only does a company need to spend significant money customizing and integrating their enterprise software solutions, but they often spend significant amounts of money on custom applications that tie into and depend on the software.

What might seem to be discrete enterprise software applications are really tangled masses of single-vendor functionality, tightly-coupled customizations and integrations, and custom code tied into this motley mess. In fact, when we ask people to describe their enterprise architecture (EA), they often point to the gnarly mess of enterprise software they purchased, customized, and maintain. That’s not EA. That’s an ugly baby only a mother could love.

Yet, companies constantly share with us their complete dependence on a handful of applications for their daily operation. Imagine what would happen at any large business if you were to shut down their single-vendor ERP, CRM, or SCM solutions. Business would grind to a halt.

While some would insist on the necessity of single-vendor, commercial enterprise software solutions as a result, we would instead assert how remarkably insane it is for companies to have such a single point of failure. Dependence on a single product, single vendor for the entirety of a company’s operations is absolutely ludicrous in an IT environment where there’s no technological reason to have such dependencies. The more you depend on one thing for your success, the less you are able to control your future. Innovation itself hangs in the balance when a company becomes so dependent on another company’s ability to innovate. And given the relentless pace of innovation, we see huge warning signs.

Services, clouds, and mashups: Why buy enterprise software?

In previous ZapFlashes, we talked about how the emergence of services at a range of disparate levels combined with evolutions in location- and platform-independent, on-demand, and variable provisioning enabled by clouds, and rich technologies to facilitate simple and rapid service composition will change the way companies conceive of, build, and manage applications.

Instead of an application as something that’s bought, customized, and integrated, the application itself is the instantaneous snapshot of how the various services are composed together to meet user needs. From this perspective, enterprise software is not what you buy, but what you do with what you have.

One outcome of this perspective on enterprise software is that companies can shift their spending from enterprise software licenses and maintenance (which eats up a significant chunk of IT budgets) to service development, consumption, and composition.

This is not just a philosophical difference. This is a real difference. While it is certainly true that services expose existing capabilities, and therefore you still need those existing capabilities when you build services, moving to SOA means that you are rewarded for exposing functionality you already have.

Whereas traditional enterprise software applications penalize legacy because of the inherent cost of integrating with it, moving to SOA inherently rewards legacy because you don’t need to build twice what you already have. In this vein, if you already have what you need because you bought it from a vendor, keep it – but don’t spend more money on that same functionality. Rather, spend money exposing and consuming it to meet new needs. This is the purview of good enterprise architecture, not good enterprise software.

When you ask these people to show you their enterprise software, they’ll simply point at their collection of Services, Cloud-based applications, and composition infrastructure.

The resultant combination of legacy service exposure, third-party service consumption, and the cloud (x-as-a-service) has motivated the thinking that if you don’t already have a single-vendor enterprise software suite, you probably don’t need one.

We’ve had first-hand experience with new companies that have started and grown operations to multiple millions of dollars without buying a penny of enterprise software. Likewise, we’ve seen billion-dollar companies dump existing enterprise software investments or start divisions and operations in new countries without extending their existing enterprise software licenses. When you ask these people to show you their enterprise software, they’ll simply point at their collection of services, cloud-based applications, and composition infrastructure.

Some might insist that cloud-based applications and so-called software-as-a-service (SaaS) applications are simply monolithic enterprise software applications deployed using someone else’s infrastructure. While that might have been the case for the application service provider (ASP) and SaaS applications of the past, that is not the case anymore. Whole ecosystems of loosely-coupled service offerings have evolved in the past decade to value-add these environments, which look more like catalogs of service capabilities and less like monolithic applications.

Want to build a website and capture lead data? No problem -- just get the right service from Salesforce.com or your provider of choice and compose it using web services or REST or your standards-based approach of choice. And you didn’t incur thousands or millions of dollars to do that.

Open source vs. commercial vs. build your own

Another trend pointing to the stalling of enterprise software growth is the emergence of open source alternatives. Companies now are flocking to solutions such as WebERP, SugarCRM Community Edition, and other no-license and no-maintenance fee solutions that provide 80% of the required functionality of commercial suites.

While some might point at the cost of support for these offerings, we point out the factor of difference between support and license/maintenance costs. At the very least, you know what you’re paying for. It’s hard to justify spending millions of dollars in license fees when you’re using 10% or less of a product’s capabilities.

Enhancing this open source value proposition is that others are building capabilities on top of those solutions and giving those solutions away as well. The very nature of open source enables creation of capabilities that further value-adds a product suite. At some point, a given open source solution reaches a tipping point where the volume of enhancements far outweighs what any commercial vendor can offer. Simply put, when a community supports an open source effort, the result can out-innovate any commercial solution.

There are now a lot of pieces and parts available that are free, cheap, or low cost that companies can assemble into not only workable, but scalable offerings that can compete with many commercial offerings.

Beyond open source, commercial, and SaaS-cum-cloud offerings, companies have a credible choice in building their own enterprise software application. There are now a lot of pieces and parts available that are free, cheap, or low cost that companies can assemble into not only workable, but scalable offerings that can compete with many commercial offerings. In much the same way that companies leveraged Microsoft’s Visual Basic to build applications using the thousands of free or cheap widgets and controls built by the legions of developers, so too are we seeing a movement to free or cheap Service widgets that can enable remarkably complex and robust applications.

The future of commercial enterprise software applications

It is not clear where commercial enterprise software applications go from here. Surely, we don’t see companies tearing out their entrenched solutions any time soon, but likewise, we don’t see much reason for expansion in enterprise software sales either.

In some ways, enterprise software has become every bit the legacy they sought to replace in mainframe applications that still exist in abundance in the enterprise. Smart enterprise software vendors realize that they have to get out of the application business altogether and focus on selling composable service widgets. These firms, however, don’t want to innovate their way out of business. As such, they don’t want to just provide the trains to get you from place to place, but they want to own the tracks as well.

The question is: Is the cost of the proprietary runtime infrastructure you are getting with those widgets worth the cost?

In many ways, this idea of enterprise software-as-a-platform is really just a shell game. Instead of spending millions on a specific application, you’re instead spending millions on an infrastructure that comes with some pre-configured widgets. The question is: Is the cost of the proprietary runtime infrastructure you are getting with those widgets worth the cost? Have you lost some measure of loose coupling in exchange for a “single throat to choke?”

Much of the enterprise software market is heading in direct collision course with middleware vendors who never wanted to enter the application market. As enterprise software vendors start seeing their runtime platform as the defensible position, they will start conflicting with EA strategies that seek to remove their single-vendor dependence.

We see this as the area of greatest tension in the next few years. Do you want to be in control of your infrastructure and have choice, or do you want to resign your infrastructure to the control of a single vendor, who might be one merger or stumble away from non-existence or irrelevance?

The ZapThink take

We hope to use this ZapFlash to call out the ridiculousness of multi-million dollar “applications” that cost millions more to customize to do a fraction of what you need. In an era of continued financial pressure, the last thing companies should do is invest more in technology conceived of in the 1970s, matured in the 1990s, and incrementally made worse since then.

The reliance on single-vendor mammoth enterprise software packages is not helping, but rather hurting the movement to loosely coupled, agile, composition-centric heterogeneous SOA. Now is the time for companies to pull up the stakes and reconsider their huge enterprise software investments in favor of the sort of real enterprise architecture that cares little about buying things en masse and customizing those solutions -- but instead to building, composing, and reusing what you need iteratively to respond to continuous change.

As if to prove a point, SAP stock recently slid almost 10% on missed earnings. Some may blame the overall state of the economy, but we point to the writing on the wall: All the enterprise software that could be sold has been sold, and the reasons for buying or implementing new licenses are few and far between. Invest in enterprise architecture over enterprise software, services over customizations, and clouds over costly and unpredictable infrastructure -- and you’ll be better off.

This guest post comes courtesy of Ronald Schmelzer, senior analyst at ZapThink.

---

SPECIAL PARTNER OFFER

SOA and EA Training, Certification,
and Networking Events

In need of vendor-neutral, architect-level SOA and EA training? ZapThink's Licensed ZapThink Architect (LZA) SOA Boot Camps provide four days of intense, hands-on architect-level SOA training and certification.

Advanced SOA architects might want to enroll in ZapThink's SOA Governance and Security training and certification courses. Or, are you just looking to network with your peers, interact with experts and pundits, and schmooze on SOA after hours? Join us at an upcoming ZapForum event. Find out more and register for these events at http://www.zapthink.com/eventreg.html.

---

Business and technical cases build for data center consolidation and modernization

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. View a full transcript or download a copy. Learn more. Sponsor: Akamai Technologies.

Data-center consolidation and modernization of IT systems helps enterprises reduce cost, cut labor, slash energy use, and become more agile.

Infrastructure advancements, standardization, performance density, and network services efficiencies are all allowing for bigger and fewer data centers and strategically architected and located facilities that can efficiently carry more of the total IT requirements load.

But to gain the benefits of these large and strategic infrastructure undertakings, the impact on the network beyond the firewall has to be considered. User expectations for performance and IT requirements for reliability need to be maintained, and even improved.

Fewer data centers means longer distances between servers and users. Network services and Internet performance management therefore need to be brought considered to produce the desired effect of topnotch applications and data delivery to enterprises, consumers, partners, and employees at far lower cost.

Here to help us better understand how to get the best of all worlds -- that is, high performance and lower total cost from data center consolidation -- we're joined by James Staten, Principal Analyst at Forrester Research; Andy Rubinson, Senior Product Marketing Manager at Akamai Technologies, and Tom Winston, Vice President of Global Technical Operations at Phase Forward, a provider of integrated data management solutions for clinical trials and drug safety. The panel is moderated by me, BriefingsDirect's Dana Gardner, principal analyst at Interarbor Solutions.

Here are some excerpts:

Staten: Oftentimes, the biggest reason to do [consolidation] is because you have sprawl in the data center. You're running out of power, you're running out of the ability to cool any more equipment, and you are running out of the ability to add new servers, as your business demands them.

If there are new applications the business wants to roll out, and you can't bring them to market, that's a significant problem. This is something the organizations have been facing for quite some time.

As a result, if they can start consolidating, they can start moving some of these workloads onto fewer systems. This allows them to reduce the amount of equipment they have to manage and the number of software licenses they have to maintain and lower their support costs. In the data center overall, they can lower their energy costs, while reducing some of the cooling required.

... Most applications actually end up consuming on average only 15-20 percent of the server. If that's the case, you've got an awful lot of headroom to put other applications on there.

We were isolating applications on their own physical systems, so that they would be protected from any faults or problems with other applications that might be on the same system and take them down. Virtualization is the primary isolating technology that allows us to do that.

... More and more applications are being broken down into modules, and, much like the web services and web applications that we see today, they're broken into tiers. Individual logic runs on its own engine, and all of that can be spread across some more monetized, consistent infrastructure. We are learning these lessons from the dot-coms of the world and now the cloud-computing providers of the world, and applying them to the enterprise.

... On average, across all the enterprises we have spoken to, you can realistically expect to see about a 20 percent cost reduction from doing this. But, as you said, if you've got 5,000 servers, and they're all running at 5 percent utilization, there are big gains to be had.

Rubinson: I focus mainly on delivery over the Internet. There are definitely some challenges, if you're talking about using the Internet with your data center infrastructure -- things like performance latency, availability challenges from cable cuts, and things of that nature, as well as security threats on the Internet.

It's thinking about how can you do this, how can you deliver to a global user base with your data center, without having to necessarily build out data centers internationally, and to be able to do that from a consolidated standpoint.

... From the cost perspective, we're able to eliminate unnecessary hardware. We're able to take some of that load off of the servers, and do the work in the cloud, which also helps reduce them.

... In terms of responsiveness, by using the Internet, you can deploy a lot more quickly. It allows us to give that same type of performance, availability, and security that you would get from having a private WAN, but doing it over the much less expensive Internet.

This is really important, as we have seen more and more users that are going outside of the corporate [networks]. People are connecting to suppliers, to partners, to customers, and to all sorts of things now.

... By optimizing the cloud, we're able to speed the delivery of information from the origin as well. That's where it's benefiting folks like Tom, where he is able to not only cache information, but the information that is dynamic, that needs to get back from the data center, goes more quickly.

Winston: When I joined [Phase Forward], it had two different data centers -- one on the East Coast and one on the West Coast. We were facing the challenge of potentially having to expand into a European data center, and even potentially a Pacific Rim data center.

By continuing to expand our virtualization efforts, as well as to leverage some of the technologies that Andy just mentioned ... Internet acceleration via some of the Akamai technologies, we were able to forgo that data center expansion. In fact, we were able to consolidate our data center to one East Coast data center, which is now our primary hosting center for all of our applications.

So it had a very significant impact for us by being able to leverage both that WAN acceleration, as well as virtualization, within our own four walls of the data center.

We run electronic data capture (EDC) software, and pharmacovigilance software for the largest pharmaceutical and clinical device makers in the world. They are truly global organizations in nature. So, we have users throughout the world, with more and more heavy population coming out of the Asia Pacific area.

... We have a very large, diverse user base that is accessing our applications 24x7x365, and, as a result, we have performance needs all the time for all of our users.

... Our primary application, our flagship application, is a product called InForm, which is the main EDC product that our customers use across the Internet. It's accelerated using Akamai technology, and almost 100 percent of our content is dynamic. It has worked extremely well.

Staten: ... Users are all over the place. Whether they are an internal employee, a customer, or a business partner, they need to get access to those applications, and they have a performance expectation that's been set by the Internet. They expect whatever applications they are interacting with will have that sort of local feel.

That's what you have to be careful about in your planning of consolidation. You can consolidate branch offices. You can consolidate down to fewer data centers. In doing so, you gain a lot of operational efficiencies, but you can potentially sacrifice performance.

You have to take the lessons that have been learned by the people who set the performance bar, the providers of Internet-based services, and ask, "How can I optimize the WAN? How can I push out content? How can I leverage solutions and networks that have this kind of intelligence to allow me to deliver that same performance level?" That's really the key thing that you have to keep in mind. Consolidation is great, but it can't be at the sacrifice of the user experience.

... The right location [for data centers] has to be optimized for a variety of factors. It has to be optimized for where the appropriate skill sets are. It has to be optimized for the geographic constraints that you may be under.

We're able to take some of that load off of the servers, and do the work in the cloud, which also helps reduce them.

You may be doing business in a country in which all of the citizen information of the people who live in that country must reside in that country. If that's the case, you don't necessarily have to own a data center there, but you absolutely have to have a presence there.

Winston: ... We had users in China who, due to the amount of traffic that had to traverse the globe, were not happy with the performance of the application. Specifically, we brought in Akamai to start with a very targeted group of users and to be able to accelerate for them the application in that region.

It literally cut the problem right out. It solved it almost immediately. At that point, we then began to spread the rest of that application acceleration product across the rest of our domains, and to continue to use that throughout the product set.

Rubinson: ... We recently commissioned a study with Forrester, looking at what is that tolerance threshold [for a page to load]. In the past it had been that people had tolerance for about four seconds. As of this latest study, it's down to two seconds. That's for business to consumer (B2C) users. What we have seen is that the business-to-business (B2B) users are even more intolerant of waiting for things.

It really has gotten to a point where you need that immediate delivery in order to drive the usage of the tools that are out there.

... Just putting yourself in the cloud doesn't mean that you're not going to have the same type of latency issues, delivering over the Internet. It's the same thing with availability in trying to reach folks who are far away from that hosted data center. So, the cloud isn't necessarily the answer. It's not a pill that you can take to fix that issue.

... For Akamai, it's really about how we're able to accelerate. How we are able to optimize the routing and the other protocols on the Internet to make that get from wherever it's hosted to a global set of end users.

We don't care about where they are. They don't have to be on the corporate, private WANs. It's really about that global reach and giving the levels of performance to actually provide an SLA. Tell me who else out there provides an SLA for delivery over the Internet? Akamai does.

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. View a full transcript or download a copy. Learn more. Sponsor: Akamai Technologies.

Separating core from context brings high returns in legacy application transformation

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. View a full transcript or download the transcript. Learn more. Sponsor: Hewlett-Packard.

---

Gain more insights into "Application Transformation: Getting to the Bottom Line" via a series of HP virtual conferences Nov. 3-5. For more on Application Transformation, and to get real time answers to your questions, register to the virtual conferences for your region:
Register here to attend the Asia Pacific event on Nov. 3.
Register here to attend the EMEA event on Nov. 4.
Register here to attend the Americas event on Nov. 5.

---

This podcast is the second in a series of three to examine Application Transformation: Getting to the Bottom Line. Through panel discussions we examine the rationale and likely returns of assessing the true role and character of legacy applications, and then further determine the paybacks from modernization.

To gain the most return on modernization projects, many enterprises are separating core from context when it comes to legacy enterprise applications and their modernization processes. As enterprises seek to cut their total IT costs, they need to identify what legacy assets are working for them and carrying their own weight, and which ones are merely hitching a high cost -- but largely unnecessary -- ride.

A widening cost and productivity division exists between older, hand-coded software assets and replacement technologies on newer, more efficient standards-based systems. Somewhere in the mix, there are also core legacy assets distinct from so-called contextal assets. There are peripheral legacy processes and tools that are costly vestiges of bygone architectures. There is legacy wheat and legacy chaff.

With us to delve deeper into the high rewards of transforming legacy enterprise applications is Steve Woods, distinguished software engineer at HP, and Paul Evans, worldwide marketing lead on Applications Transformation at HP. The discussion is moderated be me, Dana Gardner, principal analyst at Interarbor Solutions.

Here are some excerpts:

Evans: This podcast is about two types of IT assets: core and context. That whole approach to classifying business processes and their associated applications was invented by Geoffrey Moore, who wrote Crossing the Chasm, Inside the Tornado, etc.

He came up in Dealing with Darwin: How Great Companies Innovate at Every Phase of their Evolution with this notion of core and context applications. Core being those that provide the true innovation and differentiation for an organization. Those are the ones that keep your customers. Those are the ones that improve the service levels. Those are the ones that generate your money. They are really important, which is why they're called "core."

When these applications were invented to provide the core capabilities, it was 5, 10, 15, or 20 years ago. What we have to understand is that what was core 10 years ago may not be core anymore. There are ways of effectively doing it at a much different price point.

As Moore points out, organizations should be looking to build "core," because that is the unique intellectual property of the organization, and to then buy "context." They need to understand, how do I get the lowest-cost provision of something that doesn't make a huge difference to my product or service, but I need it anyway.

The "context" applications are not less important, but ... you should be looking to understand how that could be done in terms of lower-cost provisioning [of them].

Woods: [A lot of the interest in separating core and context in legacy IT applications] has to do with the pain users are going through. We have had customers who had assessments with us before, as much as a year ago, and now they're coming back and saying they want to get started and actually do something. So, a good deal of the interest is caused by the need to drive down costs.

Also, there's the realization that a lot of these tools -- extract, transform, and load (ETL) tools, enterprise application integration (EAI) tools, reporting, and business process management (BPM) -- are proving themselves now. We can't say that there is a risk in going to these tools. They realize that the strength of these tools is that they bring a lot of agility, solve skill sets issues, and make you much more responsive to the business needs of the organization.

... What I created at HP is a tool, an algorithm, that can go into any language legacy code and find the duplicate code, and not only find it, but visualize it in very compelling ways. That helps us drill down to identify what I call the unintended design. When we find these unintended designs, they lead us to ask very critical questions that are paramount to understanding how to design the transformation strategy.

... When you identify the IT elements that are not core and that could be moved out of handwritten code, you're transferring power from the developers -- say, of COBOL -- to the users of the more modern tools, like the BPM tools.

So there is always a political issue. What we try to do, when we present our findings, is to be very objective. You can't argue that we found that 65 percent of the application is not doing core. You can then focus the conversation on something more productive. What do we do with this? The worst thing you could possibly do is take a million lines of COBOL that's generating reports and rewrite that in Java or C# hard-written code.

We take the concept of core versus context not just to a possible off-the-shelf application, but at architectural component level. In many cases, we find that this is helpful for them to identify legacy code that could be moved very incrementally to these new architectures.

... A typical COBOL application -- this is true of all legacy code, but particularly mainframe legacy code -- can be as much as 5, 10, or 15 million lines of code. I think the sheer idea of the size of the application is an impediment. There is some sort of inertia there. An object at rest tends to stay at rest, and it's been at rest for years, sometimes 30 years.

So, the biggest impediment is the belief that it's just too big and complex to move and it's even too big and complex to understand. Our approach is a very lightweight process, where we go in and answer to a lot of questions, remove a lot of uncertainty, and give them some very powerful visualizations and understanding of the source code and what their options are.

... When you go to the legacy side of the house, you start finding that 65 percent of this application is just doing ETL. It's just parsing files and putting them into databases. Why don't you replace that with a tool? The big resistance there is that, if we replace it with a tool, then the people who are maintaining the application right now are either going to have to learn that tool or they're not going to have a job.

If we get the facts on the table, particularly visually, then we find that we get a lot of consensus. It may be partial consensus, but it's consensus nonetheless, and we open up the possibilities and different options, rather than just continuing to move through with hand-written code.

If you look at this whole core-context thing, at the moment, organizations are still in survival mode.

Evans: If you look at this whole core-context thing, at the moment, organizations are still in survival mode. Money is still tight in terms of consumer spending. Money is still tight in terms of company spending. Therefore, you're in this position where keeping your customers or trying to get new customers is absolutely fundamental for staying alive. And, you do that by improving service levels, improving your services, and improving your product.

... The line-of-business people are now pushing on technology and saying, "You can't back off. You can't not give us what we want. We have to have this ability to innovate and differentiate, because that way we will keep our customers and we will keep this organization alive."

That applies equally to the public and private sectors. The public sector organizations have this mandate of improving service, whether it's in healthcare, insurance, tax, or whatever. So all of these commitments are being made and people have to deliver on them, albeit that the money, the IT budget behind it, is shrinking or has shrunk.

The leaders must understand what drives their company. Understand the values, the differentiation, and the innovations that you want and put your money on those and then find a way of dramatically reducing the amount of money you spend on the contextual stuff, which is pure productivity.

Woods: ... Decentralizing the architecture improves your efficiency and your redundancy. There is much more opportunity for building a solid, maintainable architecture than there would be if you kept a sort of monolithic approach that's typical on the mainframe.

... The problem is sometimes not nearly as big as it seems. If you look at the analogy of the clone codes that we find, and all the different areas that we can look at the code and say that it may not be as relevant to a transformation process as you think it is.

The subject matter experts and the stakeholders very slowly start to understand that this is actually possible. It's not as big as we thought.

I do this presentation called "Honey I Shrunk the Mainframe." If you start looking at these different aspects between the clone code and what I call the asymmetrical transformation from handwritten code to model driven architecture, you start looking at these different things. You start really seeing it.

We see this, when we go in to do the workshops. The subject matter experts and the stakeholders very slowly start to understand that this is actually possible. It's not as big as we thought. There are ways to transform it that we didn't realize, and we can do this incrementally. We don't have to do it all at once.

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. View a full transcript or download the transcript. Learn more. Sponsor: Hewlett-Packard.

---

Gain more insights into "Application Transformation: Getting to the Bottom Line" via a series of HP virtual conferences Nov. 3-5. For more on Application Transformation, and to get real time answers to your questions, register to the virtual conferences for your region:
Register here to attend the Asia Pacific event on Nov. 3.
Register here to attend the EMEA event on Nov. 4.
Register here to attend the Americas event on Nov. 5.

---