Dana Gardner's BriefingsDirect: Cloud and SaaS force a rethinking of integration and middleware as services for services

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Read a full transcript or download a copy. Sponsor: Workday.

Major trends around cloud, mobile, and software as a service (SaaS) are dramatically changing the requirements and benefits of application integration.

In many respects, the emphasis now on building hybrid business processes from a variety of far-flung sources forces a rethinking of integration and middleware. Integration capabilities themselves often need to be services in order to support a growing universe of internal and external constituent business process component services.

And increasingly, integration needs to be ingrained in applications services, with the costs and complexity hidden. This means that more people can exploit and leverage integration, without being integration technology experts. It means that the applications providers are also the integration providers. It also means the stand-alone integration technology supplier business -- and those that buy from it -- are facing a new reality.

Here to explore the new era of integration-as-a-service and what it means for the future is David Clarke, Director of Integration at SaaS ERP provider Workday. He is interviewed by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: Workday is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Clarke: I remember when we originally became part of Workday several years ago, we were doing some sort of product planning and strategic thinking about how we were going to integrate the product lines and position them going forward. One of the things we had in our roadmap at the time was this idea of an [integration] appliance. So we said, "Look, we can envision the future, where all the integration is done in the cloud, but we frankly think it's like a long way off. We think that it's some years off."

... We thought the world wasn’t going to be ready soon enough to put the integration technology and stack in the cloud as well.
It just became clearer and clearer to us that there was an appetite and a willingness in our customer and prospect base to use this technology in the cloud.

Happily that turned out to have been incorrect. Over the course of the ensuing 12 months, it just became clearer and clearer to us that there was an appetite and a willingness in our customer and prospect base to use this technology in the cloud.

We never really went ahead with that appliance concept, it didn’t get productized. We never used it. We don’t need to use it. And now, as I have conversations with customers and with prospects, it just is not an issue.

In terms of it being any kind of philosophical or in principle difficulty or challenge, it has just gone away. It totally surprised me, because I expected it to happen, but thought it would take a lot longer to get to where it has got to already.

Gardner: We see that a “consumerization” of IT is taking place, where the end-users want IT in the enterprise to work as well and in the same manner as it does for their personal lives. How does that shift the thinking of an enterprise architect?

Clarke: Superficially, enterprise architects are under a lot of pressure to present technologies in ways that are more familiar to customers from their personal lives. The most specific example of that is the embrace of mobile technologies. This isn't a huge surprise. It's been a pretty consistent pattern over a number of years that workforce mobility is a major influence on product requirements.

Mobile devices

We've seen that very significant proportions of access to our system is via mobile devices. That informs our planning and our system architecture. We're invested heavily in mobile technologies -- iPad, Android, BlackBerry, and other clients. In my experience, that’s something that's new, with the customer enterprise architects. This is something they have to articulate, defend, and embrace.

Historically, they would have been more concerned with the core issues of scalability, reliability, and availability. Now, they've got more time to think about these things, because we as SaaS vendors have taken a lot of things that they used to do off of their plates.

Historically, a lot of time was spent by enterprise architects worrying about the scalability and reliability of the enterprise application deployments that they had, and now that’s gone away. They get a much higher service level agreement (SLA) than they ever managed to operate by themselves when they run their own systems.

So, while they have different and new things to think about because of the cloud and mobility, they also have more head space or latitude to do that, because we have taken some of the pain that they used to have away.

Gardner: I suppose that as implications pan out around these issues, there will be a shift in economics as well, whereby you would pay separately and perhaps on a capital and then operating basis for integration.
They also have more head space or latitude to do that, because we have taken some of the pain that they used to have away from them.

If integration by companies like Workday becomes part-and-parcel of the application services -- and you pay for it on an operating basis only -- how do traditional business models and economics around middleware and integration survive?

Clarke: I'd certainly hate to be out there trying to sell middleware offerings stand-alone right now, and clearly there have been visible consolidations in this space. I mentioned BEA earlier as being the standard bearer of the enterprise Java generation of middleware that’s been acquired by Oracle.

They are essentially part of the application stack, and I'm sure they still sell and license stand-alone middleware. Obviously, the Oracle solutions are all on-premise, so they're still doing on-premise stuff at that level. But, I would imagine that the economics of the BEA offering is folded very much into the economics of the Oracle application offering.

In the web services generation of middleware and integration, which essentially came after the enterprise Java tier, and then before the SOA tier, there was a pretty rapid commoditization. So, this phenomenon was already starting to happen, even before the cloud economics were fully in play.

Then, there was essentially an increased dependence or relevance of open source technologies -- Spring, JackBe, free stacks -- that enabled integration to happen. That commoditization was already starting to happen.

Open source pressure

So even before the advent of the cloud and the clear economic pressure that put on stand-alone integration, there was already a separate pressure that was originating from open source. Those two things together have, in my view, made it pretty difficult to sustain or to conceive a sustainable integration model.

A lot of the investment dollars that have gone into something like integration market are now going elsewhere in infrastructure. They're going into storage. They're going into availability. They're going certainly to cloud platforms. It would need to be a brave venture capitalist now who would write a check to a company coming in with a bright idea for a new on-premise middleware stack. So that business is gone.

... Workday is an applications company. We're an on-demand apps company and we build and serve human capital management (HCM), financials, and enterprise resource planning (ERP) application suites.

Cape Clear, which was my former company, was acquired by Workday about three years ago. We were partners, but as Workday’s business expanded significantly, they saw that providing a compelling and a differentiated integration experience in the context of this new cloud architecture was going to be something that was very important to them. So they acquired Cape Clear and we became part of the overall Workday organization.

Gardner: So there seem to be two fundamental things going on here. One, is taking integration to the on-demand or SaaS domain, but second, there is also this embedding integration functionality into the application.
One of the perpetual holy grails of the middleware industry, when it was a stand-alone undertaking was to find a way to express and expose middleware and integration concepts in a way that they could be used by mere mortals.

How does someone like yourself who is crafting the middleware integration capabilities need to shift their thinking in order to go “to the cloud,” but also become part-and-parcel with the application?

Clarke: One of the perpetual holy grails of the middleware industry, when it was a stand-alone undertaking, was to find a way to express and expose middleware and integration concepts in a way that they could be used by mere mortals, by business analysts, by people who weren't necessarily very deep technologists with deep technology expertise.

In my experience, the middleware industry never achieved that. So, they didn't really ever find a metaphor or a use model that enabled less skilled, but nonetheless technically savvy, people to use their products.

As you observe in the applications game, you absolutely have to get there, because fundamentally what you're doing here is you are enabling companies and individuals to solve business problems and application problems. The integration arises as a necessity of that or as a consequence of that. In and of itself, it isn't useful.

Designing applications

The most specific thing that we've seen is how we build, manipulate, and use extremely sophisticated integration technology. We spend a lot of our time thinking about how to design that into the application, so that it can be experienced and consumed by users of the application who don’t know anything about XML, Java, web protocols, or anything like that.

... Business analysts can very easily and visually define what they are getting and putting it in terms of the business concepts and the business objects they understand. They can define very simple transformations, for example, going from a payroll input to a check, or going from a report of absences by departments to a payroll input.

They're consuming and using integration technologies in a very natural way in the context of their day-to-day working in the web layer in these systems. They're not programmers. They're not developers. They're not thinking about it that way.

It's quite empowering for the teams that we have had working on this technology to see if it's usable in that way by the business analysts here. It's the closest I've seen people get to capturing this unicorn of enabling integration technology to be actually used by business people.
It's the closest I've seen people get to capturing this unicorn of enabling integration technology to be actually used by business people.

Gardner: Do you think in 10 years, or maybe 5, we won’t even be thinking about integration? It will really be a service, a cloud service, and perhaps it will evolve to be a community approach.

Clarke: I'll make two main observations in this area.

First, there is an important difference between a general-purpose platform or integration platform and then a more specific one, which is centered around a particular application domain. Workday is about the latter.

We're building a very powerful set of cloud technologies, including an integration cloud or an integration platform in the cloud, but it’s very focused on connecting essentially to and from Workday, and making that very easy from a variety of places and to a variety of places.

What we're not trying to create is a general-purpose platform, an associated marketplace, in the way that maybe somebody like Salesforce.com is doing with AppExchange or Google with App Engine for app development. In a sense, our scope is narrower in that way, and that’s just how we're choosing to prosecute the opportunity, because it’s harder to establish a very horizontal platform and it’s just difficult to do.

I referred earlier to the problem that middleware companies traditionally have of doing everything and nothing. When you have a purely horizontal platform that can offer any integration or any application, it’s difficult to see exactly which ones are going to be the ones that get it going.

The way we're doing this is therefore more specific. We have a similar set of technologies and so on, but we're really basing it very much around the use case that we see for Workday. It’s very grounded in benefits integrations, payroll integrations, financial integrations, payment integrations. And every one of our deployments has tens, dozens, hundreds of these integrations. We're constantly building very significant volume, very significant usage, and very significant experience.

Developing marketplace

I can see that developing into a marketplace in a limited way around some of those key areas and possibly broadening from there.

That's one of the interesting areas of distinction between the strategies of the platform vendors as to how expansive their vision is. Obviously expansive visions are interesting and creating horizontal platforms is interesting, but it’s more speculative, it’s riskier, and it takes a long time. We are more on the specific side of that.

You mentioned collaborating and how this area of business processes and people collaborating in the community. I referred earlier to this idea that we're focusing on these key use cases. What’s arising from those key use cases is a relatively small set of documents and document formats that are common to these problem areas.

Lately, I've been reading, or rereading, some of the RosettaNet stuff. RosettaNet has been around forever. It was originally created in the early '80s. As you know, it was essentially a set of documents, standard documents, interchange formats for the semiconductor or the technology manufacturing industry, and it has been very successful, not very prominent or popular, but very successful.

What we see is something similar to RosettaNet starting to happen in the application domain where, when you are dealing with payroll providers, there is a certain core set of data that gets sent around. We have integrated to many dozens of them and we have abstracted that into a core documentary that reflects the set of information and how it needs to be formatted and how it needs to be processed.
These are very good vectors for cooperation and for collaboration around integrations, and they're a good locus around which communities can develop standardized documents.

In fact, we now have a couple of payroll partners who are directly consuming that payroll format from us. So, in the same way that there are certain HR XML standards for benefits data, we can see other ones emerging in other areas of the application space.

These are very good vectors for cooperation and for collaboration around integrations, and they're a good locus around which communities can develop standardized documents, which is the basis for integration. That’s intriguing to me, because it all derives from that very specific set of use cases that I just never really saw as a general-purpose integration vendor.

Gardner: What Workday is supporting is an applications-level benefit. A business process, like a network, is perhaps more valuable as the number of participants in the process increases, and become able to participate with a fairly low level of complexity and friction.

So, if we make this shift to a Metcalfe's Law-type of "the more participants, the more valuable it is to all of those participants," shouldn’t we expect a little bit of a different world around integration in the next few years?

Business process

Clarke: That’s right. We don’t really see or envisage this very transactional marketplace, where you just have people buying a round of maps or integrations and installing them. We see it happening in the context of a business process.

For example, hiring. As somebody hires somebody into Workday, there are typically many integration points in that business process -- background checking, provisioning of security cards, and creation of email accounts. There is a whole set of integration points. We're increasingly looking to enable third parties to easily plug-in into those integration points in a small way, for provisioning an email account, or in a big way, like managing a whole payroll process.

It’s that idea of these integrations as being just touch points and jumping-off points from an overall business process, which is quite a different vision from writing cool, stand-alone apps that you can then find and store from inside of our platform marketplace.

It’s that idea of an extended business process where the partners and partner ISVs and customers can collaborate very easily, and not just at install time or provisioning time, but also when these processes are running and things go wrong, if things fail or errors arise.

You also need a very integrated exception-handling process, so that customers can rapidly diagnose and correct these errors when they arise. Then, they have a feeling of being in a consistent environment and not like a feeling of having 20 or 30 totally unrelated applications executing that don’t collaborate and don’t know about each other and aren’t executing the context within the same business process. We're keen to make that experience seamless.
You also need a very integrated exception handling process, so that customers can rapidly diagnose and correct these errors when they arise.

Gardner: I can also see where there is a high incentive for the participants in a supply chain or a value chain of some sort to make integration work. Do you already see that the perception of cooperation for integration is at a different plane? Where do you expect that to go?

Clarke: Totally, already. Increasingly -- pick an area, but let's say for learning management or something -- if we integrate, or if multiple people integrate to us or from us, then customers already are starting to expect that those integrations exist.

Now they're starting to ask about how good they are, what's the nature of them, what SLAs can they expect here? The customers are presuming that an integration, certainly between Workday and some other cloud-based service, either exists already or is very easy and doable.

But they're looking through that, because they're taking the integration technology level questions for granted. They're saying, "Given that I can make such an integration work, how is it really going to work, what's the SLA, what happens if things go wrong, what happens when things fail?"

What's really interesting to me is that customers are increasingly sophisticated about exploring the edge cases, which they have seen happen before and have heard about them before. They're coming to us upfront and saying, "What happens if I have issues when my payroll runs? Who do I go to? How do you manage that? How do you guys work with each other?"

Consistent information

We, therefore, are learning from our customers and we're going to our ISV and services partners, like our payroll partners, our learning management partners, our background checking partners and saying, "Here is the contract that our customers expect. Here is the service that they expect." They're going to ask us and we want to be able to say that this partner tests against every single update and every single revision of the Workday software. They will handle a seamless support process where you call one number and you get a consistent set of information.

Customers are really looking through the mere fact of a technical integration existing and asking about what is my experience going to be and actually using this day-to-day across 50 geographies and across population of 20,000 employees. I want that to be easy.

It’s a testament to the increasing sophistication of the integration technology that people can take that for granted. But as I say, it’s having these increasingly interesting and downstream effects in terms of what people are expecting from the business experience of using these integration systems in the context of a composite business process that extend beyond just one company.

Gardner: When you have a massive, complex, integration landscape, does it makes sense to focus on the application provider as that point of responsibility and authority? Or does it have to be federated?

Clarke: What we are gradually feeling our way toward here is that for us that’s the central concept of this federation of companies. We think obviously of Workday being in the middle of that. It depends on what your perspective is, but you have this federation of companies collaborating to provide the service ultimately, and the question is, where do they choke?

And it's not realistic to say that you can always come to Workday, because if we are integrating to a payroll system on behalf of somebody else, and we correctly start off and run the payroll or send the payroll requests, and then there is an error at the other end, the error is happening ultimately in the other payroll engine. We can't debug that. We can't look at what happened. We don't necessarily even know what the data is.
As we run any integration in our cloud, there is a very consistent set of diagnostics, reporting, metrics, error handling, error tracking that is generated and that's consistent.

We need a consistent experience for the customer and how that gets supported and diagnosed. Specifically, what it means for us today is that, as we run any integration in our cloud, there is a very consistent set of diagnostics, reporting, metrics, error handling, error tracking that is generated and that's consistent across the many types of integrations that we run.

Again, as our partners become more savvy at working with us, and they know more about that, they can then more consistently offer resolution and support to the customers in the context of the overall Workday support process.

For us, it’s really a way of building this extended and consistent network of support capability and of trust. Where customers have consistent experiences, they have consistent expectations around how and when they get support.

The most frustrating thing is when you are calling one company and they're telling you to call the other company, and there isn’t any consistency or it’s hard to get to the bottom of that. We're hopeful that enlightened integrations around business processes, between collaborating companies, as I have described, will help me to get some of that.

... The technology is important, but it's not enough. People just don't just want technology. They want well-intentioned and an honest collaboration between their vendors to help them do the stuff efficiently.

... [Technically] there’s nothing fundamentally new, in some sense. I've worked in several generations of integration and middleware technology, and each one is a refinement of the past, and you're standing on the shoulders of giants, to badly paraphrase Newton.

Packaged and presented

A lot of the underlying technology you're using for integration, a lot of the underlying concepts, are not that new. It's just the way that they're being packaged and presented. In some cases, it's the protocols that we're using, and certainly some of the use models. But the ways you're accessing them and consuming them are different. So, it is in that sense [this is] evolutionary.

Gardner: While you have put quite a bit of emphasis on the tool side in order to make this something that mere mortals can adjust and operate, you've also done a lot of heavy lifting on the connections side. You recognized that in order to be successful with an integration platform, you had to find the means in which to integrate to a vast variety of different types of technologies, services, data, and so forth. Tell me what you've done, not only on the usability, but on the applicability across a growing universe of connection points.

Clarke: That’s another interesting area. As you say, there are thousands or millions of different types of endpoints out there. This being software, it can map any data format to any other data format, but that’s a trivial and uninformative statement, because it doesn’t help you get a specific job done.

Essentially what we've been trying to do is identify categories of target systems and target processes that we need to integrate with and try to optimize and focus our efforts on that.

For example, pretty much the majority of our customers have a need to integrate to and from benefit systems for 401(k), healthcare, dental, visual plans, and so forth. It's an extremely common use case. But, there is still a wide diversity of benefits providers and a wide variety of formats that they use.

We've studied the multiple hundreds of those benefits providers that we've experienced by working with our customers and we've abstracted out the most common format scenarios, data structures, and so forth, and we have built that into our integration layer.

Configure your data set

You can very easily and rapidly and without programming configure your specific data set, so that it can be mapped into and out of your specific set of benefits providers, without needing to write any code or build a custom integration.

We've done that domain analysis in a variety of areas, including but not limited to benefits. We've done it for payroll and for certain kinds of financial categories as well. That's what's enabling us to do this in a scalable and repeatable way, because we don’t want to just give people a raw set of tools and say, "Here, use these to map anything to anything else." It's just not a good experience for the users.