Making Hadoop safe for 'clusterophobics'

This guest post comes courtesy of Tony Baer's OnStrategies blog. Tony is senior analyst at Ovum.

By Tony Baer

Hadoop remains a difficult platform for most enterprises to master. For now skills are still hard to come by – both for data architect or engineer, and especially for data scientists. It still takes too much skill, tape, and baling wire to get a Hadoop cluster together. Not every enterprise is Google or Facebook, with armies of software engineers that they can throw at a problem. With some exceptions, most enterprises don’t deal with data on the scale of Google or Facebook either – but the bar is rising.

If 2011 was the year that the big IT data warehouse and analytic platform brand names discovered Hadoop, 2012 becomes the year where a tooling ecosystem starts emerging to make Hadoop more consumable for the enterprise. Let’s amend that – along with tools, Hadoop must also become a first-class citizen with enterprise IT infrastructure. Hadoop won’t cross over to the enterprise if it has to be treated as some special island. That means meshing with the practices and technology approaches that enterprises are using to manage their data centers or cloud deployments. Like SQL, data integration, virtualization, storage strategy, and so on.

Admittedly, much of this cuts against the grain of early Hadoop deployment that stressed open source and commodity infrastructure. Early adopters did so out of necessity as commercial software ran out of gas for Facebook when its data warehouse daily refreshes were breaking terabyte range, not to mention that the cost of commercial licenses for such scaled out analytic platforms wouldn’t have been trivial. Anyway, Hadoop’s linearity leverages scale out of commodity blades and direct attached disk as far as the eye can see, enabling such an almost pure noncommercial approach. At the time, Google’s, Yahoo’s, and Facebook’s issues were considered rather unique – most enterprise don’t run global search engines – not to mention that their business was built on armies of software engineers.

Something's got to give

As we’ve previously noted, something’s got to give on the skills front. Hadoop in the enterprise faces limits – the data problems are getting bigger and more complex for sure, but resources and skills are far more finite. So we envision tools and solutions addressing two areas:

1. Products that address “clusterophobia” – organizations that seeks the scalable analytics of Hadoop but lack the appetite to erect infinite data centers out in the fields or hire the necessary skillsets. Obviously, using the cloud is one option – but the questions there revolve around whether corporate policies allow maintenance of data off premises, and also, as data store size grows, whether the cloud is still economical.
   

2. The other side of the coin is consummability – tools that simplify access to and manipulation of the data.

In the run-up to this year’s Hadoop Summit, a number of tooling announcements addressing clusterophobia and consumption are pouring out.

The workloads are going to get more equitably distributed, and in the long run, we wouldn’t be surprised to see more Hadoop-only appliances.

On the fear of clusters side, players like Oracle, EMC Greenplum, and Teradata Aster are already offering appliances that simplify deployment of Hadoop, typically in conjunction with an Advanced SQL analytic platform. While most vendors position this as a way for Hadoop to “extend’ your data warehouse so you perform exploration in Hadoop, but the serious analytics in SQL, we view appliances as more than transitional strategy. The workloads are going to get more equitably distributed, and in the long run, we wouldn’t be surprised to see more Hadoop-only appliances, sort of like Oracle’s (for the record, they also bundle another NoSQL database).

Also addressing the same constituency are storage and virtualization – facts of life in the data center. For Hadoop to cross over to the enterprise, it, too, must get virtualization-friendly. Storage is an open question. The need for virtualization becomes even more apparent because (1) the exploratory nature of Hadoop analytics demands the ability to try out queries offline without having to disrupt or physically build a new cluster; and (2) the variable nature of Hadoop processing suggests that workloads are likely to be elastic. So we’ve been waiting for VMware to make their move. VMware – also part of EMC – has announced a pair of initiatives. First, they are working with the Apache Hadoop project to make the core pieces (HDFS and MapReduce) virtualization-aware, and separately, they are hosting their own open source project (Serengeti) for virtualizing Hadoop clusters. While Project Serengeti is not VM-specific, there’s little doubt that this will be a VMware project (we’d be shocked if the Xen folks were to buy in).

Storage follows

Where there’s virtualized servers, storage often closely follows. A few months back, EMC dropped the other shoe, finally unveiling a strategy for leveraging Isilon with the Greenplum HD platform, the closest thing in NAS that replicates the scale-out model storage model popularized with Hadoop. This opens an argument of whether the scales of data in Hadoop make premium products such as Isilon unaffordable. The flip side however is the “open source tax,” where you hire the skills in your IT organization to manage and deploy scale-out storage, or pay consultants to do it for you.

In the spirit of making Hadoop more consummable, we expect a lot of vibes from new players that are simplifying navigation of Hadoop and building SQL bridges. Datameer is bringing down the pricing of its uber Hadoop spreadsheet to personal and workgroup levels courtesy of entry level pricing from $299 to $2999. Teradata Aster, which already offers a patented framework that translates SQL to MapReduce (there are also others out there) is now taking an early bet on the incubating Apache HCatalog metadata spec so that you could write SQL statements that go up against Hadoop. It joins approaches such as those from Hadapt, which hangs SQL tables from HDFS file nodes, and mainstream BI players such as Jaspersoft, that already provide translators that can grab reports directly from Hadoop.

In the spirit of making Hadoop more consummable, we expect a lot of vibes from new players that are simplifying navigation of Hadoop and building SQL bridges.

This doesn’t take away from the evolution of the Hadoop platform itself. Cloudera and Hortonworks are among those releasing new distributions that bundle their own mix of recent and current Apache Hadoop modules. While the Apache project has addressed the NameNode HA issue, it is still early in the game with bringing enterprise-grade manageability to MapReduce. That’s largely an academic issue as the bulk of enterprises have yet to implement Hadoop. By the time enterprises are ready, many of the core issues should resolve — although there will always be questions about the uptake of peripheral Hadoop projects.

What’s more important – and where the action will be – is in tools that allow enterprises to run and, more importantly, consume Hadoop. A chicken and egg situation, enterprises won’t implement before tools are available and vice versa.

This guest post comes courtesy of Tony Baer's OnStrategies blog. Tony is senior analyst at Ovum.

You may also be interested in:

• Fast data hits the big data fast lane
  
• EMC's Hadoop strategy cuts to the chase
• Big data consolidation race enters home stretch, as Teradata buys Aster Data
• Oracle fills another gap in its big data offering
• VMforce: Cloud mates with Java marriage of necessity for VMware and Salesforce.com
• HP buys Fortify, and it's about time

Cloud-powered services deliver new revenue and core business agility for SMB travel insurance provider Seven Corners

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy. Sponsor: VMware.

The latest BriefingsDirect cloud computing discussion centers on how small-to-medium sized business (SMB) Seven Corners, a travel insurance provider in Indiana, created and implemented an agile and revenue-generating approach to cloud services.

Seven Corners went beyond the typical efficiency and cost conservation benefits of cloud to build innovative business services that generate whole new revenue streams. A VMware-enabled cloud infrastructure allowed Seven Corners to rapidly reengineer its IT capabilities and spawn a new vision for its agility and future growth.

Here to share their story on an SMB's journey to cloud-based business development is George Reed, CIO of Seven Corners Inc., based in Carmel, Indiana. The interview is conducted by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: VMware is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Gardner: I often hear that culture will trump strategy. It sounds as if in your organization -- maybe because you're an SMB and you can get the full buy-in of your leadership -- you actually were able to make culture into the strategy?

Reed: Absolutely! By changing the culture and getting the departments out there to ask, "Is this stuff you're doing going to help me with this problem?" "Well, yes, it will," and then you deliver on that promise.

When you make a promise and you deliver on it, on or ahead of schedule and under budget people begin to believe, they're willing to participate and actively suggest other possible uses with technology that maybe you didn't think of. So you end up with a great technology-business relationship, which had the immediate result for the owners who were out looking to buy an insurance services application or rent one.

They said, "We're a very entrepreneurial company with so many different lines of business that there is nothing out there that would really work for us. We believe in your IT. Build us one." This year we rolled out an application called Access that is so configurable you could run any kind of insurance services through it, whether you're insuring parrots, cars, people, trucks, or whatever.

When you make a promise and you deliver on it, on or ahead of schedule and under budget people begin to believe.

Gardner: Let's learn some more about that. One of the nice things about early successes is that you get that buy-in and the cultural adoption, but you've also set expectations for ongoing success. I suppose it's important to keep the ball rolling and to show more demonstrable benefits.

So when it came to not only repaving those cow paths, making them more efficient, cutting cost, delivering that six-month return on investment, what did you enable? What did you then move forward to to actually create new business development and therefore new revenue?

Reed: By continuing to lower IT cost, when we virtualized the desktops using VMware's View, and then VMware's Horizon which makes it device-independent, it’s easier for everybody to work. That had appreciable productivity improvements out in the departments.

At the same time, my apps development group began designing and building an application called AXIS. What this came out of was that when we went to insurance conventions, talked to carriers and asked, "What are the top 10 reasons you want to fire your third-party administrator today?"

Technology was always part of those top 10 answers. So we devised and developed an application that would eliminate those as problems. The result is that this year, since February, we have four insurance carriers that were working with either their own stuff or third-party administrator, big COBOL mainframe monsters that are just so spaghetti-coded and heavy you can never really get out of it.

Already implemented

They see what our tool is doing and they ask these questions. "What are the specs for me to be able to connect to it?" "Well, you have to have an Internet connection and something smarter than a coffee cup." "That’s it?" "Yeah, that’s it." "Well, what’s the price for us to implement your solution?" "None. "It’s already implemented. You just import your business."

The jaws drop around the table. "How will I be able to see my data?" "You’ll all get in and look at it." "You mean I don’t ask for a report?" "You can, but it’s easier if you just log and look at your report."

They're flocking in. The biggest challenge is keeping up with the pace of the growing business and that goes back to planning for the future. I planned a storage solution and a compute solution. I can just keep adding blades and adding trays of storage without any outage at all.

Gardner: Pay as you go?

Reed: Yes, and the neat thing is that that the process of closing transactions will run about $7 million in revenue a year. It will cost about $1.5 million to service that revenue. Not a bad profit base for an SMB. And it’s because we're going to come in at 45 percent less than their existing service provider, and we're going to provide services that are 100 times better.

Gardner: Before we learn more about that approach and process, perhaps you could explain for our listener’s benefit what Seven Corners is, how large you are, what you do, and just describe what you are doing as a business.

Reed: Seven Corners started in 1993 as Specialty Risk International, and as we began to grow around the globe with customers in every time zone there is, the company changed its name to Seven Corners.

It started out providing specialty travel insurance, trip cancellation insurance, then began providing third-party administrator, general insurance services, and emergency assistance services around the globe. We have about 800 programs in five major product lines to span hundreds of thousands of members.

The company itself is about 170-175 people. We've been enjoying double-digit growth every year. As a matter of fact, I believe that at the end of February they hit the double-digit growth goal for 2012. So we're going to exceed that as the year goes on. You are going to see the technology has driven some of that growth.

Gardner: Who do you consider your primary customers? Is it travel agencies, or do you go direct to the travelers themselves, or a mixture?

If you communicate that in a passionate, effective way with the ownership and the executive group, you come out of the room with authority to move forward.

Reed: About 50 percent of the business is online. You go to the website to fill out a form to figure out what you need. You buy it right then and there, collect your virtual ID card, and you're on your way.

We have customers that are high-tech companies who are sending their people all over the world. They'll buy, at the corporate level, trip cancellation, trip assistance, and trip major medical insurance.

Then, there are universities and other affinity groups. They have students traveling abroad. We have companies sending people to work in the United States. Then, we are doing benefit management and travel assistance for numerous government agencies, US Department of State, Bureau of Prisons, AmeriCorps, and the Peace Corps as well.

Gardner: So if I understand correctly, George, you're saying that you went from being a broker of services, finding insurance carrier services, and then packaging and delivering them to end users, to now actually packaging insurance as a service. You're packaging the ability to conduct business online and packaging that, in addition, to the value-added services for insurance. Does that capture what’s happened?

With a solid, virtual, private-cloud solution, the cost of delivering technology services is just very low per-member serviced.

Reed: It does, and providing immediate access to what any stakeholder in that insurance lifecycle needs improves the quality of the end product. It lowers the cost of the healthcare.

We're starting to get into the state Medicaid benefits management as well. We're saying, "You're spending too much." The first slide in the proposal is always, "You're spending too much on Medicaid healthcare. We're going to help you cut it down and we are going to do it right now." You get attention, when you just walk in bold as brass and say that.

With a solid, virtual, private-cloud solution, the cost of delivering technology services is just very low per member service. In insurance, there are only so many ways to improve profit. One is to grow business. We all know that. But, two is to reduce the time and price of processing a claim, reduce the time and price to implement new business and collect the premium.

We’ve built an infrastructure and now an application platform that does those things. In the old system, the time to process a claim around here was about 30 minutes going through a complex travel medical claim with tons of lines. Now it’s about 15 seconds.

Gardner: This is really fascinating. It strikes me that you’ve sort of defined the future of business. Being an early adopter of technologies that make you agile and efficient means that you're not only passing along the ability to be productive in your traditional business, but you’ve moved into an adjacency that allows you to then take away from your partners and customers the processes that they can’t do as well and embed those into the services that you provide.

When, of course, you can charge back to them at a rate that was lower for them in the first place. You can really grow your definition of being a business within your market.

Think big

Reed: That’s correct, and you can do this in any industry. There is a talk that I’ve given a couple of times at Butler University about how you can never stop being small, until you think big. You have to say, "What would it take for me to do that? Everything is on the table. What would it take?"

My boss does that to me and my direct reports as well. "What would it take for us to accomplish this thing by this time? Don’t worry about what it is. Just tell me what it would take. Let’s see, if we can’t do it." That’s the philosophy that this company was built on.

By the end of the year, we're not only going to be doing all that kind of service for carriers, but we are going to stand up an instance of AXIS to be software as a service and every small third-party administrator (TPA) in the country is going to have an opportunity to buy seats at this servicing application that is easily configurable to whatever their business rules are.

Gardner: When you began this journey to transform how Seven Corners does IT, did you have a guiding principle or vision? Was there a stake in the ground that you could steer toward?

Reed: I did. I was brought in specifically to be an innovative change agent to take them from where they were to where they wanted to get as a business. They just weren’t there at the time. My vision was to come in, stop the bleeding, pick off the low hanging fruit to step up to the next level, and then build a strategic road map that would not only meet -- but exceed -- the needs of the business, and reach out 5-10 years beyond.

Gardner: Is there anything specifically about an SMB that you think enabled such agility? I know it’s very difficult in large companies to make such a change in short order. Do you have a certain advantage being smaller?


Authority to move

Reed: You do. If you're in a privately held SMB, your goal is to identify a problem or an opportunity, categorize what it would cost to resolve it or achieve it, and show the return on investment (ROI). If you communicate that in a passionate, effective way with the ownership and the executive group, you come out of the room with authority to move forward. That’s exactly what I did.

Gardner: And on one side of your business equation, of course, you have these consumers and customers, but you also must have quite a variety of partners, other insurance carriers, for example, medical insurance providers, and so forth. So you need to match and broker services among and between all these?

Multiple carriers

Reed: Correct. We have multiple carriers and do some of the advances around Seven Corners. We’ve got about four more carriers starting to move business our way. So you have to meet all of their needs, reporting needs, timeliness of service, and support their customers. At the same time, we've got all the individuals and groups that we're doing business with and we are doing it across five different revenue-producing lines of business.

Gardner: Let's move back to what it is that you've done at an architecture level. As you had that vision about what you needed, and as you gathered requirements in order to satisfy these business needs, what did you look for and what did you start to put in place?

Reed: The first thing I did is assess what was going on in the server room. On my first day, walking in there and looking around, I saw a bunch of oversized Dell desktops that were buffed up to be servers. There were about 140 of those in there.

I was thinking, "This is 2000-2003 technology. I'm here in 2010. This isn't going to work." It was an archaic system that was headed to failure, and that was one of the reasons they knew they had to change. They could no longer sustain either the applications or the hardware itself.

What I wanted to do was put in an infrastructure that would completely replace what was there. The company had grown to the point where there was so much transactional volume, so many thousands of people hitting the member portals. The cloud started to speak to me. I needed to be serving member portals out on a private cloud. I needed to be reaching out to the 15,000 medical providers around the world that we're talking with to get their claims without them sending paper or emails.

It was an archaic system that was headed to failure, and that was one of the reasons they knew they had to change.

I looked at an integrating partner locally in the Midwest. It's called Netech. I said, "Here is my problem. I know that within four months my major servers that are backing up or providing our insurance applications are going to fail. You can't even get parts on eBay for them anymore. I need you to come back to me in a week with a recommendation on how you understand my problem, what you recommend I do about it, and what it's going to cost, wheels-on, out the door."

Gardner: Just to be clear, did you have a certain level of virtualization already in place at this point?

Reed: No, there was nothing virtual in the building. It was all physical. Netech went away and came back a week later, after looking at the needs and asking a ton of questions, as any good partner would do. They said, "Here's what we think you need to do. You need something that's expandable easily for your compute side. We recommend Cisco UCS. Here is a plan for that.

"You need storage that can provide secure multitenancy, because you've got a lot of different carriers that don't want their information shared. They want to know that it's very secured. We recommend NetApp’s FlexPod solution for that.

"And for your virtualization, hub and going to the cloud, we're seeing the best results with VMware's product."

Then, we started with VMware Enterprise, and when it became available, upgraded to vSphere 5.0.

Up and running

They came in with a price, so I knew exactly what it would cost to implement, and they said, they could do it in three months. I went to the owners and said, "You're losing $100,000 revenue a month because of this situation in your server room. You'll pay for this entire project in six months." They said, "Well, get it done." And so we launched. In about two and a half months we were up and running. Our partnership with Netech has had a dramatic impact on speed-to-production for each phase of our virtualization.

Gardner: When you looked at creating a private-cloud fabric to support your application, were these including your internal back-office types of apps? Did you have ERP and communications infrastructure and apps that you needed to support? Clearly, you talked about portals and being able to create Web services and integrate across the business processes, all the above. Did you want to put everything in this cloud or did you segment?

Reed: I wanted to get off the old analog phone system that was there and go to a Cisco Unified Communications Manager, which is a perfect thing to drop into a virtual environment. I wanted to get everybody on the voice-over-IP (VOIP) phones. I wanted to get my call center truly managing 24×7×365, no matter where they were sitting.

I wanted to get users, both customer users, partner users and then the people from Seven Corners to get to where it didn't matter what they were connecting to the Internet with. They could connect to my system and see their data and it would never leave my server, which is one of the beauties of a private cloud, because the data never leaves a secure environment.

Gardner: Did you get a vision to bring all of your apps into this or did you want to segment, sort of was this a crawl-walk-run approach to bringing your apps into this cloud or was this more of a transformation, even shock therapy, to kind of do it all at once to get it done?

That got everybody thinking, "Hey, IT can deliver."

Reed: The server virtualization was a shock therapy, because the infrastructure was very outdated, and any piece of it failing is a failure. It doesn’t matter which one it was.

So we took a 144 servers virtual and took all the storage into the NetApp controller, achieving an immediate 50 percent de-duplication rate. And the efficiency in spinning up servers for a development group to support them, was such that we were cutting a ton of manpower that was required to spin those up. Instead of 4-5 days to set up a server for them to work on a new application, it's now 4-5 minutes.

In the first three days here, inside IT and out in the business, I said, "I need a list by Friday, please, of the top five things we need to keep doing, stop doing, or start doing."

I got great input and then I picked the pain points. That's what I call the low-hanging fruit. We knocked those out the first month, just general technology support. That got everybody thinking, "Hey, IT can deliver." Originally they had a nickname for the department --"The Island of Dr. No." ... No, we can't do this, no, we can't do that.

Getting champions

We said, "Let's find a way to say, "Yes," or at least offer a different solution." When we killed some of those early problems, we ended up getting champions out of opposition. It became very easy to get the company to do business differently, and to put up with the testing, user acceptance process, and training to use different technology services.

We're running on vSphere 5.0 and have put in a vCenter Configuration Manager and Operations Manager. We're doing our virtual desktops using the power of ThinApp and VMware Horizon.

Then to make the cloud come to being we got the vCloud Director and vShield in. We're doing a lot of business with the government, and with government agencies we have to be Federal Information Security Management Act (FISMA) compliant which makes HIPAA compliance look kind of easy.

The other technology that the VMware is living on is the Cisco UCS, and it’s all being stored on NetApp FlexPod with data replication. In a few months, it will be live mirror for both compute and the data.

Mobile devices

Gardner: And how does that now set you up for perhaps moving toward the use of mobile devices? Clearly, you've got some of those interface issues resolved by going fully virtual. Is there a path to allowing choice, even bring your own device (BYOD) types of choice by your users going to new classes of devices?

Reed: We’re working on the BYOD program now. A lot of the department heads have been issued devices through our secure wireless in the building. A couple of them have iPads and a couple of them have Android OSes. Several of us with the new Cisco phone systems have the Cisco tablet that's your actual VoIP phone station and your thin client.

To get ready to go to a meeting, I get off the phone, pull the tablet out of the docking station, and into the meeting. I have my desktop right there. I've never logged off. When I need to go home for the night, I take it home, and log in through my wireless. I have my Voice over IP handset, and I'm calling from my desk phone from anywhere in the world.

So we're already doing what I would call a pilot program to prove it out to everybody and get them used to it. Right now, our sales guys love the fact that they just pull that thing out of the docking station and go off to show a client what our software and services really are.

Gardner: That's a really impressive story, George, and you've been able to do this in just a couple of years. It’s really astonishing. Before we close out, could you provide some advice to other SMBs that have heard your story and can see the light bulbs for their own benefits going off in their heads? Do you have any advice in hindsight from your experience that you would share with them in terms of getting started?

Reed: The key is that you can’t get to where you’re going if you don’t set the vision of what you want to be able to do. To do that, you have to assess where you’re at and what the problems are.

Phase your solutions that you’re going to recommend to solve big problems early and get buy-in. And when you’ve got executive buy-in, you have department heads and users buying in, it’s easy to get a lot of stuff done very quickly, because people aren’t resisting the change.

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy. Sponsor: VMware.

You may also be interested in:

• Case Study: Strategic Approach to Disaster Recovery and Data Lifecycle Management Pays Off for Australia's SAI Global
• Virtualization Simplifies Disaster Recovery for Insurance Broker Myron Steves While Delivering Efficiency and Agility Gains Too
• SAP Runs VMware to Provision Virtual Machines to Support Complex Training Courses
• Case Study: How SEGA Europe Uses VMware to Standardize Cloud Environment for Globally Distributed Game Development
• Germany's Largest Travel Agency Starts a Virtual Journey to Get Branch Office IT Under Control
• Virtualized Desktops Spur Use of 'Bring You Own Device' in Schools, Allowing Always-On Access to Education Resources