How Etsy uses big data for ecommerce to put buyers and sellers in the best light

The next BriefingsDirect big data case study discussion explores how Etsy, a global e-commerce site focused on handmade and vintage items, uses data science to improve buyers and sellers’ discovery and shopping experiences.

We'll learn how mining big data at speed and volume helps Etsy define and distribute top trends, and allows those with specific interests to find items that will best appeal to them.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy.

To learn more about leveraging big data in the e-commerce space, please join Chris Bohn aka “CB,” a Senior Data Engineer at Etsy, based in Brooklyn, New York. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: Tell us about Etsy for those that aren’t familiar with it. I've heard it described as it’s like being able to go through your grandmother's basement. Is that fair?

CB: Well, I hope it’s not as musty and dusty as my grandmother’s basement. The best way to describe it is that Etsy is a marketplace. We create a marketplace for sellers of handcrafted goods and the people who want to buy those goods.

We've been around for 10 years. We're the leader in this space and we went public in 2015. Just some quick little metrics. The total of value of the merchandise sold on Etsy in 2014 was about $1.93 billion. We have about 1.5 million sellers and about 22 million buyers.

Gardner: That's an awful lot of stuff that’s being moved around. What does the big data and analytics role bring to the table?

CB: It’s all about understanding more about our customers, both buyers and sellers. We want to know more about them and make the buying experience easier for them. We want them to be able to find products easier. Too much choice sometimes is no choice. You want to get them to the product they want to buy as quickly as possible.

We also want to know how people are different in their shopping habits across the geography of the world. There are some people in different countries that transact differently than we do here in the States, and big data lets us get some insight into that.

Gardner: Is this insight derived primarily from what they do via their clickstreams, what they're doing online? Or are there other ways that you can determine insights that then you can share among yourself and also back to your users?

Data architecture

CB: I'll describe our data architecture a little bit. When Etsy started out, we had a monolithic Postgres database and we threw everything in there. We had listings, users, sellers, buyers, conversations, and forums. It was all in there, but we outgrew that really quickly, and so the solution to that was to shard horizontally.

CB

Now we have many hundreds of sharded MySQL servers, horizontal. Then we decided that we needed to do some analytics on this stuff. So we scratched our heads. This was about five years ago. So we said, "Let’s just set up a Postgres server and we'll copy all the data from these shards into the Postgres server that we call BI server." And we got that done.

Then, we kind of scratched our heads and said, "Wait a minute. We just came full circle. We started with a monolithic database, then we went sharded, and now all the data is back monolithic."

It didn't perform well, because it's hard to get the volume of big data into that database. A relational database like Postgres just isn’t designed to do analytic-type queries. Those are big aggregations, and Postgres, even though it is a great relational database, is really tailored for single-record lookup.

So we decided to get something else going on. About three-and-a-half years ago, we set about searching for the replacement to our monolithic business-intelligence (BI) database and looked at what the landscape was. There were a number of very worthy products out there, but we eventually settled on HPE Vertica for a number of reasons.

One of those is that it derives, in large part, from Postgres. Postgres has a Berkeley license. So  companies could take it private. They can take that code and they don’t have to republish it out to the community, unlike other types of open source copyright agreements.

So we found out that the parser was right out of Postgres and all the date handling and typecasting stuff that is usually different from database to database was exactly spot-on the same between Vertica and Postgres. Also, data ingestion via the copy command is the best way to bulk-load data, exactly the same in both, and it’s the same format.

There were a number of very worthy products out there, but we eventually settled on Vertica for a number of reasons.

We said, "This looks good, because we can get the data in quickly, and queries will probably not have to be edited much." So that's where we went. We experimented with it and we found exactly that. Queries would run unchanged, except they ran a lot faster and we were able to get the data in easily.

We built some data replication tools to get data from the shards and also some legacy Postgres databases that we had laying around for billing and got that all data into HPE Vertica.

Then, we built some tools that allowed our analysts to bring over custom tables they had created on that old BI machine. We were able to get up to speed really quickly with Vertica, and boom, we had an analytics database that we were able to hit the ground running with it.

Gardner: And is the challenge for you about the variety of that data? Is it about the velocity that you need to move it in and out? Is it about simply volume that you just have so much of it, or a little of some of those?

All of the above

CB: It’s really all of those problems. Velocity-wise, we want our replication system to be eventually consistent, and we want it to be as near real-time as possible. There is a challenge in that, because you really start to get into micro-batching data in.

This is where we ended up having to pay off some technical debt, because years ago, disk storage was fairly pricey, and databases were designed to minimize storage. Practices grew up around that fact. So data would get deleted and updated. That's the policy that the early originators of Etsy followed when they designed the first database for it.

Start Your
HPE Vertica
Community Edition Trial Now

Eventually what we have got now is lossy data. If someone changes the description or the tags that are associated with a listing, the old ones go away. They are lost forever. And that's too bad, because if we kept those, we can do analytics on a product that wasn’t selling for a long time and all of a sudden it started selling. What changed? We would love to do analytics on that, but we can't do it because of the loss of data. That's one thing that we learned in this whole process.

But getting back to your question here about velocity and then also the volume of data, we have a lot of data from our production databases. We need to get it all into Vertica. We also have a lot of clickstream data. Etsy is a top 50 website, I believe, for traffic, and that generates a lot of clicks and that all gets put into Vertica.

This is where we ended up having to pay off some technical debt, because years ago, disk storage was fairly pricey, and databases were designed to minimize storage.

We run big batch jobs every night to load that. It's important that we have that, because one of the biggest things that our analytics like to do is correlate clickstream data with our production data. Clickstream data doesn't have a lot of information about the user who is doing those clicks. It’s just information about their path through the site at that time.

To really get a value-add on that, you want to be able to join on your user details tables, so that you can know where this person lives, how old they are, or their buying history in the past. You need to be able to join those, too, and we do that in HPE Vertica.

Gardner: CB, give us a sense about the paybacks, when you do this well, when you've architected, and when you've paid your technical debts, as you put it. How are your analysts able to leverage this in order to make your business better and make the experience of your users better?

CB: When we first installed Vertica, it was just a small group of analysts that were using it. Our analytics program was fairly new, but it just exploded. Everybody started to jump in on it, because all of a sudden, there was a database with which you could write good SQL, with a rich SQL engine, and get fantastic results quickly.

The results weren’t that different from what we were getting in the past, but they were just coming to us so fast, the cycle of getting information was greatly shortened. Getting result sets was so much better that it was like a whole different world. It’s like the Pony Express versus email. That’s the kind of difference it was. So everybody started jumping in on it.

More dashboards

Engineers who were adding new facets of the product wanted to have dashboards, more or less real time, so they could monitor what the thing was doing. For example, we added postage to Etsy, so that our sellers can have preprinted labels. We'd like to monitor that in real time to see how it's this going. Is it going well or what?

That was something that took a long time to analyze before we got into big-data analytics. All of a sudden, we had Vertica and we could do that for them, and that pattern has repeated with other groups in the company.

We're doing different aspects of the site. All of a sudden, you have your marketing people, your finance people, saying, "Wow, I can run these financial reports that used to take days in literally seconds." There was a lot of demand. Etsy has about 750 employees and we have way more than 200 Vertica accounts. That shows you how popular it is.

Start Your
HPE Vertica
Community Edition Trial Now

One anecdotal story. I've been wanting to update Vertica for the past couple of months. The woman who runs our analytics team said, "Don't you dare. I have to run Q2 numbers. Everybody is working on this stuff. You have to wait until this certain week to be able to do that." It’s not just HPE Vertica, but big data is now relied on for so many things in the company.

Gardner: So the technology led to the culture. Many times we think it's the other way around, but having that ability to do those easy SQL queries and get information opened up people's imagination, but it sounds like it has gone beyond that. You have a data-driven company now.

CB: That's an astute observation. You're right. This is technology that has driven the culture. It's really changed the way people do their job at Etsy. And I hear that elsewhere also, just talking to other companies and stuff. It really has been impactful.

This is technology that has driven the culture. It's really changed the way people do their job at Etsy.

Gardner: Just for the sake of those of our readers who are on the operations side, how do you support your data infrastructure? Are you thinking about cloud? Are you on-prem? Are you split between different data centers? How does that work?

CB: I have some interesting data points there for you. Five-plus years ago, we started doing Hadoop stuff, and we started out spinning up Hadoop in Amazon Web Service (AWS).

We would run nightly jobs. We collected all of the search terms that were used and buying patterns and we fed these into MapReduce jobs. The output from that then went into MATLAB, and we would get a set of rules out of that, that then would drive our search engine, basically improving search.

Commodity hardware

We did that for a while and then realized we were spending a lot of money in AWS. It was many thousands of dollars a month. We said, "Wait a minute. This is crazy. We could actually buy our own servers. This is commodity hardware that this can run on, and we can run this in our own data center. We will get the data in faster, because there are bigger pipes." So that's what we did.

We created what we call Etsydoop, which has got 200+ nodes and we actually save a lot of money doing it that way. That's how we got into it.

We really have a bifurcated data analytics, big-data system. On the one hand, we have Vertica for doing ad hoc queries, because the analysts and the people out there understand SQL and they demand it. But for batch jobs, Hadoop rocks, and it's really, really good for that.

But the tradeoff is that those are hard jobs to write. Even a good engineer is not going to get it right every time, and for most analysts, it's probably a little bit beyond their reach to get down, roll up their sleeves, and get into actual coding and that kind of stuff.

The analysts and the people out there understand SQL and they demand it. But for batch jobs, Hadoop rocks, and it's really, really good for that.

But they're great at SQL, and we want to encourage exploration and discovering new things. We've discovered things about our business just by some of these analysts wildcatting in the database, finding interesting stuff, and then exploring it, and we want to encourage that. That's really important.

Gardner: CB, in getting to understand Etsy a little bit more, I saw that you have something called Top Trends and Etsy Finds, ways that you can help people with affinity for a product or a craft or some interest to pursue that. Did that come about as a result of these technologies that you have put in place, or did they have a set of requirements that they wanted to be able to do this and then went after you to try to accommodate it? How do you pull off that Etsy Finds capability?

CB: A lot of that is cross-architecture. Some of our production data is used to find that. Then, a lot of the hard crunching is done in Vertica to find that. Some of it is MapReduce. There's a whole mix of things that go into that.

I couldn't claim for Etsy Finds, for example, that it’s all big data. There are other things that go in there, but definitely HPE Vertica plays a role in that stuff.

I'll give you another example, fraud. We fingerprint a lot of our users digitally, because we have problems with resellers. These are people who are selling resold mass-produced stuff on Etsy. It's not huge, but it's an annoyance. Those products compete against really quality handmade products that our regular sellers sell in their shops.

Sometimes it’s like a game of Whack-a-Mole. You knock one of these guys down -- sometimes they're from the Far East or other parts of the world -- and as soon as you knock one down, another one pops up. Being able to capture them quickly is really important, and we use Vertica for that. We have a team that works just on that problem.

What's next?

Gardner: Thinking about the future, with this great architecture, with your ability to do things like fraud detection and affinity correlations, what's next? What can you do that will help make Etsy more impactful in its market and make your users more engaged?

CB: The whole idea behind databases and computing in general is just making things faster. When the first punch-card machines came out in the 1930s or whatever, the phone companies could do faster billing, because billing was just getting out of control. That’s where the roots of IBM lie.

As time went by, punch cards were slow and they wanted to go faster. So they developed magnetic tape, and then spinning rust disks. Now, we're into SSDs, the flash drives. And it’s the same way with databases and getting answers. You always want to get answers faster.

We do a lot of A/B testing. We have the ability to set the site so that maybe a small percentage of users get an A path through the site, and the others a B path, and there's control stuff on that. We analyze those results. This is how we test to see if this kind of button work better than this other one. Is the placement right? If we just skip this page, is it easier for someone to buy something?

The whole idea behind databases and computing in general is just making things faster.

So we do A/B testing. In the past, we've done it where we had to run the test, gather the data, and then comb through it manually. But now with Vertica, the turnaround time to iterate over each cycle of an A/B test has shrunk dramatically. We get our data from the clickstreams, which go into Vertica, and then the next day, we can run the A/B test results on that.

The next step is shrinking that even more. One of the themes that’s out there at the various big data conferences is streaming analytics. That's a really big thing. There is a new database out there called PipelineDB, a fork of Postgres. It allows you to create an event steam into Postgres.

You can then create a view and a window on top of that stream. Then you can pump your event data, like your clickstream data, and you can join the data in that window to your regular Postgres tables, which is really great, because we could get A/B information in real time. You set up a one minute turnaround as opposed to one day. I think that’s where a lot of things are going.

If you just look at the history of big data, MapReduce started about 10 years ago at Google, and that was batch jobs, overnight runs. Then, we started getting into the columnar stores to make databases like Vertica possible, and it’s really great for aggregation. That kicked it up to the next level.

Another thing is real-time analytics. It’s not going to replace any of these things, just like Vertica didn't replace Hadoop. They're complementary. Real-time streaming analytics will be complementary. So we're continuing to add these tools to our big data toolbox.

Gardner: It has compressed those feedback loops if we provide that capability into innovative, creative organization. The technology might drive the culture, and who knows what sort of benefits they will derive from that.

All plugged in

CB: That's very true. You touched earlier about how we do our infrastructure. I'm in data engineering, and we're responsible for making sure that our big databases are healthy and running right. But we also have our operations department. They're working on the actual pipes and hardware and making sure it’s all plugged in. It's tough to get all this stuff working right, but if you have the right people, it can happen.

I mentioned earlier about AWS. The reason we were able to move off of that and save money is because we have the people who can do it. When you start using AWS extensively, what you're doing is you are paying for a very high priced but good IT staff at Amazon. If you have got a good IT staff of your own, you're probably going to be able to realize some efficiencies there, and that's why really we moved over. We do it all ourselves.

Gardner: Having it as a core competency might be an important thing moving forward. The whole idea behind databases and computing in general is just making things faster.

CB: Absolutely. You have to stay on top of all this stuff. A lot is made of the word disruption, and you don't go knocking on disruption’s door; it usually knocks on yours. And you had better be agile enough to respond to it.

I'll give you an example that ties back into big data. One of the most disruptive things that has happened to Etsy is the rise of the smartphone. When Etsy started back in 2005, the iPhone wasn't around yet; it was still two years out. Then, it came on the scene, and people realized that this was a suitable device for commerce.

Start Your
HPE Vertica
Community Edition Trial Now

It’s very easy to just be complacent and oblivious to new technologies sneaking up on you. But we started seeing that there was more and more commerce being done on smartphones. We actually fell a little bit behind, as a lot of companies did five years ago. But our management made decisions to invest in mobile, and now 60 percent of our traffic is on mobile. That's turned around in the past two years and it has been pretty amazing.

Big data helps us with that, because we do a lot of crunching of what these mobile devices are doing. Mobile is not the best device maybe for buying stuff because of the form factor, but it is a really good device for managing your store, paying your Etsy bill, and doing that kind of stuff. So we analyzed all that and crunched it in big data.

Gardner: And big data allowed you to know when to make that strategic move and then take advantage of it?

CB: Exactly. There are all sorts of crossover points that happen with technology, and you have to monitor it. You have to understand your business really well to see when certain vectors are happening. If you can pick up on those, you're going to be okay.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy. Sponsor: Hewlett Packard Enterprise.

You may also be interested in:

• IoT plus big data analytics translate into better services management at Auckland Transport
• How HPE’s internal DevOps paved the way for speed in global software delivery
• Extreme Apps approach to analysis makes on-site retail experience king again
• How New York Genome Center Manages the Massive Data Generated from DNA Sequencing
• The UNIX evolution: A history of innovation reaches an unprecedented 20-year milestone
• Redmonk analysts on best navigating the tricky path to DevOps adoption
• DevOps by design--A practical guide to effectively ushering DevOps into any organization
• Need for Fast Analytics in Healthcare Spurs Sogeti Converged Solutions Partnership Model
• HPE's composable infrastructure sets stage for hybrid market brokering role
• Nottingham Trent University Elevates Big Data's role to Improving Student Retention in Higher Education
• Forrester analyst Kurt Bittner on the inevitability of DevOps
• Agile on fire: IT enters the new era of 'continuous' everything

The UNIX evolution: A history of innovation reaches an unprecedented 20-year milestone

The next BriefingsDirect expert panel discussion examines the illustrious 20-year history of the UNIX operating system environment as an industry-wide and global standard success story.

It's not often that you reach a multi-decade anniversary in information technology, especially where the technology's relevance remains so high and the promise of more innovation and value is so needed and promising.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or  download a copy.

To chart this unprecedented journey of interoperable software and stewardship success, we're joined by a distinguished panel: Andrew Josey, Director of Standards at The Open Group; Darrin Johnson, Director of Solaris Engineering at Oracle; Tom Mathews, distinguished engineer of Power Systems at IBM, and Jeff Kyle, Director of Mission-Critical Solutions at Hewlett Packard Enterprise. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: UNIX has been evolving during probably the most dynamic time in business and technology. How is it that UNIX remains so prominent, a standard that has clung to its roots, with ongoing compatibility and interoperability? How has it been able to maintain its relevance in such a dynamic world?

Josey: Thank you, Dana. As you know UNIX was started in Bell Labs by Ken Thompson and Dennis Ritchie back in 1969. It was a very innovative, a very different approach, an approach that has endured over time. We're seeing, during that time, a lot of work going on in different standards bodies.

Josey

We saw, in the early '80s, the UNIX wars, almost different fractured versions, different versions of the operating system, many of them incompatible with each other and then the standards bodies bringing them together.

We saw efforts such as the IEEE POSIX, and then X/OPEN. Later, The Open Group was formed to bring that all together when the different vendors realized the benefits of building a standard platform on which you can innovate.

So, over time, the standards have added more and more common interfaces, raising the bar upon which you can place that innovation. Over time, we've seen changes like in the mid-'90s, when there was a shift from 32-bit to 64 bit computing.

At that time, people asked, "How will we do that? Will we do it the same way?" So the UNIX vendors came to what, at that time, was X/OPEN. We had an initiative called the Large File Summit and we agreed the common way to do that. That was a very smooth transition.

Today, everybody takes it for granted that the UNIX systems are scalable, powerful, and reliable, and this is all built on that 64-bit platform, and multi-processor, and all these capabilities.

That's where we're seeing the standards come in allowing the philosophy, the enduring, adaptable pace, and that’s the UNIX platform that's relevant today. We're saying it is today’s virtualization, cloud, and big data, which is also driven by UNIX systems in the back office.

The Open Group involvement

Gardner: So while we're looking at UNIX’s 40-year history, we're focusing on the 20-year anniversary of the single UNIX specification and the ability to certify against that, and that’s what The Open Group has been involved in, right?

Josey: We were given the UNIX trademark from Novell back in, I think it was 1993, and at that point, the major vendors came together to agree on a common specification. At the time, its code name was Spec 1170. There were actually 1168 interfaces in the Spec, but we wanted to round up and, apparently, that was also the amount of money that they spent at the dinner after they completed the spec.

So, we adopted that specification and we have been running certification programs against that.

Gardner: Darrin, with the dynamic nature of our industry now -- with cloud, hybrid cloud, mobile, and a tightening between development and operations -- how is it that UNIX remains relevant, given these things that no one really saw coming 20 years ago?

Johnson: I think I can speak for everybody here that all our companies provide cloud services, whether it’s public cloud, private cloud, or hybrid cloud, and whether it’s infrastructure as a service (IaaS), software as a service (SaaS), or any of the other as a service options. The interesting thing is that to really be able to provide that consistency and that capability to our customers, we rely on a foundation -- and that foundation is UNIX.

Johnson

So our customers, even though they can maybe start with IBM, have choice. In turn, from a company perspective, instead of having to reinvent the wheel all the time for the customer or for our own internal development, it allows us to focus on the value-add, the services, the capabilities that build upon that foundation of UNIX.

So, something that may be 20 years old, or actually 40 years from the original version of UNIX, has evolved with such a solid foundation that we can innovate on.

Gardner: And what’s the common thread around that relevance? Is it the fact that it is consistently certified, that you have assurance that what's running in one place will run into another on any hardware? How is it that the common spec has been so instrumental in making this a powerful underpinning for so much modern technology?

Josey: A solid foundation is built upon standards, because we can have, like you mentioned, assurance. If you look at the certification process, there are more than 45,000 test cases that give assurance to developers, to customers that there's going to be determinism. All of the IT people that I have talked to say that a deterministic behavior is critical, because when it’s non-deterministic, things go wrong. Having that assurance enables us to focus on what sits on top of it, rather than does the ‘ls’ command work right or can we know how much space is in a file system. Those are givens. We can focus on the innovation instead.

Gardner: Over the past decades, UNIX has found itself at the highest echelon of high-performance computing, in high-performance cloud environments. Then, it goes down to the desktop as well as into mobile devices, pervasively, and as micro-devices, embedded and real-time computing. How has that also benefited from standards, that you have a common code base up and down the spectrum, from micro to macro?

Several components

Johnson: If you look at the standard, it contains several components, and it's really modular in a way that, depending on your need, you can pick a piece of it and support that. Maybe you don't need the complete operating system for a highly scalable environment. Maybe you just need a micro-controller. You can pick the standard, so there is consistency at that level, and then that feeds into the development environment in which an engineer may be developing something.

That scales. Let’s say you need a lot of other services in a large data center where you still have that consisting throughout. Whether it’s Solaris, AIX, HP-UX, Linux, or even FreeBSD, there's a consistency because of those elements of the standard.

Gardner: Developers are, of course, essentially making any platform continue over time, the chicken and the egg relationship, the more apps the more relevant the platform, the stronger and more pervasive the platform the more likely the apps. So, Jeff, for developers, what are some of the primary benefits of UNIX and how has that contributed to its longevity?

Kyle: As was said for developers, it’s the consistency that really matters. UNIX standards develop and deliver consistency. As we look at this, we talk about consistent APIs, consistent command line, and consistent integration between users and applications.

Kyle

This allows the developers to focus a lot more on interesting challenges and customer value at the application and user level. They don’t have to focus so much on interoperability issues between OSes or even interoperability issues between versions of the single OS. Developers can easily support multiple architectures in heterogeneous environments, and in today’s virtualized cloud-ready world, it’s critical.

Gardner: And while we talk about the past story with UNIX, there's a lot of runway to the future. Developers are now looking at issues around mobile development, cloud-first development. How is UNIX playing a role there?

Kyle: The development that’s coming out of all of our organizations and more organizations is focused first on cloud. It’s focused first on fully virtualized environment. It’s not just the interoperability with applications, but it is the interoperability between, as I said before, the heterogeneous environments, the multiple architectures.

In the end, customers are still trying to do the same things that they always have. They're trying to use applications in technology to get data from one place to another and more effectively and efficiently use that data to make business decisions. That’s happening more and more "mobile-y," right?

I think every HP-UX, AIX, Solaris, and UNIX system out there is fully connected to a mobile world and the Internet of Things (IoT). We're securing it more than any customers realize.

Gardner: Tom, let’s talk a little bit about the hardware side and the ability to recognize that cost and risk have a huge part of decision-making for customers, for enterprises. What is it about UNIX now, and into the future, that allows a hardware approach that keeps those cost risks down, that makes that a powerful combination for platform?

Scale up

Mathews: The hardware approach for the UNIX has traditionally been scale-up. There are a lot of virtues and customer values around scale-up. It’s a much simpler environment to administer, versus the scale-out environment that’s going to have a lot more components and complexity. So that’s a big value.

Mathews

The other core value that is important to many of our customers is that there has been a very strong focus on reliability, availability, and scalability. At the end of the day, those three words are very important to our customers. I know that they're important to the people that run our systems, because having those values allows them to sleep right at night and have weekends with their families and so forth. In addition to just running the business, things have to stay up -- and it has been that way for a long time, 7×24×365.

So these three elements -- reliability and availability and scalability -- have been a big focus, and a lot of that has been delivered through the hardware environment, and in addition to the standards.

The other thing that is critical, and this is really a very important area where the standards figure in, is around investment protection. Our customers make investments in middleware and applications and they can’t afford to re-gen those investments continuously as they move through generations of operating systems and so forth.

The standards play into that significantly. They provide the stable environment. In the standards test suite right now, there are something like 45,000 tests for testing for standards. So it's stability, reliability, availability, and serviceability in this investment-protection element.

Gardner: Now, we've looked at UNIX through the lens of developers, hardware, and also performance and risk. But another thing that people might not appreciate is a close relationship between UNIX and the advancement of the Internet and the World Wide Web. The very first web servers were primarily UNIX. It was the de-facto standard. And then service providers, those folks hosting websites were hosting the Internet itself, were using UNIX for performance and reliability reasons.

Any standard, whether it’s Ethernet or UNIX, helps bring things together in a way that you don’t have to think about how to get data from one point to another.

So, Darrin, tell us about the network side of this. Why has UNIX been so prevalent along the way when the high-performance networks, and then the very important performance characteristics of a web environment, came to bear?

Johnson: Again, it’s about the interconnectedness. Back in my younger years, having to interface Ethernet with AppleTalk, with picking your various technologies, just the interfacing took so much time and effort.

Any standard, whether it’s Ethernet or UNIX, helps bring things together in a way that you don’t have to think about how to get data from one point to another. Mobility really is about moving data from one place to another in a quick fashion where you can do transactions in microseconds, milliseconds, or seconds. You want some assurance in the data that you send from one place to another. But it's also about making sure of, and this is a topic that’s really important today, security.

Knowing that when you have data going from one point to another point, it's secured and on each node, or each point, security continues, and so standards and making sure that IBM interacts with Oracle, interacts with HPE, really assures our customers. And the people that don’t even see the transactions going on, they can have some level of confidence that they're going to have reliable, high-performance, and secure networks.

Standardization certification

Gardner: Well, let’s dig a little bit into this notion of standardization certification, of putting things through their conformity paces. Some folks might be impatient going through that. They want to just get out there with the technology and use it, but a level of discipline and making sure that things work well can bear great fruit for those who are willing to go through that process.

Andrew, tell us about the standard process and how that’s changed over the past 20 years, perhaps to not only continue that legacy of interoperability, but perhaps also increase the speed and the usability of the standards process itself.

Josey: Since then, we've made quite a few changes in the way that we're doing the standards development ourselves. It used to be that a group of us would meet behind closed doors in different locations, and there were three of such groups of standard developers.

There was an IEEE group, an X/Open (later to become an Open Group group), and an International Standards Group. Often, they were same people who had to keep going to these same meetings, and seeing the same people but wearing different hats. As I said, it was very much behind closed doors.

As it got toward the end of the 1990s, people were starting to say that we were spending too much money doing the same thing, basically producing a pile of standards that were very similar but different. So in late 1997-1998, we formed something that we call the Austin Group.

It was basically The Open Group’s members. Sun, IBM, and HP came to The Open Group at that time, and said, "Look, we have to go and talk to IEEE, we have to talk to ISO about bringing all the experts together in a single place to do the standard. So starting in 1998, we met in Austin, at the IBM facility -- hence the name The Austin Group -- and we started on that road.

We do everything virtually and we've adopted some of the approaches of open source projects.

Since then, we developed a single set of books. On the front cover, we stamped the designation of it being an IEEE standard, an Open Group standard, or an International Standard. So technical folks only have to go to a single place, do the work once, and then we put it through the adoption processes of the individual organizations.

As we got into the new millennium, we changed our way as well. We don’t physically go and meet anywhere, anymore. We do everything virtually and we've adopted some of the approaches of open source projects, for example an open bug tracker (MantisBT).

Anybody can access the bug tracker file, file a bug against the standard and see all the comments that go in against a bug, so we are completely transparent. With the Austin Group, we allow anybody to participate. You don't have to be a member of IEEE or an international delegate any more to participate.

We've had lot of input and continue to have a lot of input from the open-source community. We've had prominent members of Linux and Open Source communities such as maintainers of key subsystems such as glibc command and utilities. They would come to us because they want to get involved, they see the value in standards.

They want to come to a common agreement on how the shell should work, how this utility should work, how they can pull POSIX threads and things into their environments, how they can find those edge cases. We also had innovation from Linux coming into the standard.

In the mid-2000s, we started to look at and say that new APIs in Linux should also be in UNIX. So in the mid-2000s, we added, I think, four specifications that we developed based on Linux interfaces from the GNU Project. So in the areas of internationalization and common APIs, that’s one thing we have always wanted to do is to look at raising that bar of common functionality.

Linux and open-source systems are very much working with the standard as much as anybody else.

Process and mechanics

Johnson: There's something I’d like to add about the process and the mechanics, because in my organization I own it. There are a couple of key points. One is, it’s great that we have an organization like The Open Group that not only helps create the standard or manage the standard, but is also developing the test suites for certification. So it’s one organization working with the community, Austin Group, and of course IEEE and The Open Group members to create a test certification suite.

If anyone of our organizations had to create or manage that separately, that’s a huge expense. They do that for them, that’s part of the service, and they have evolved that and it’s grown. I don’t know what it was originally, but 45,000 tests have grown, and they’ve made it more efficient in terms of the process. And it’s a collaborative process. If we have  an issue, is it our issues, is it the test read issue. There's a great responsiveness.

So kudos to The Open Group, because they make it easy for us to certify, that’s really our obligation to get into that discipline, but if we factor it into the typical quality assurance process as we release the operating system, whether it’s an update or a patch, or whatever, then it just becomes pretty obvious. The next major release that you want to certify, you've done most of the heavy lifting. Again, The Open Group makes it really easy to do that.

It’s that the standards have actually encouraged innovation in the software industry because that just made it easier for developers to develop, and it's less costly for them to provide their stuff across the broad range of platforms.

Mathews: Another element that’s important on this cost point is goes back to the standards and the cost of doing development. Imagine being a software ISV. Imagine a world where there were no standards. That world existed at one point in time. What that caused is this, ISVs had to spend significant effort to port their to each platform.

This is because the interfaces and the capabilities on all of those platforms will be different. You will see difference all of the way across. Now with the standards, of course, ISVs basically develop for only one platform: the platform defined by the standards.

So that’s been crucial. It’s that the standards have actually encouraged innovation in the software industry because that just made it easier for developers to develop, and it's less costly for them to provide their stuff across the broad range of platforms.

So that’s been crucial. We have three people from the major UNIX vendors on the panel, but there are other players there, too, and the standards have been critical over time for everybody, particularly when the UNIX market was made up of a lot of vendors.

Gardner: So we understand the value of standards and we see the role that a neutral third-party can play to keep those standards on track and moving rapidly. Are there some lessons from UNIX of the past 20 years that we can apply to some of the new areas where standards are newly needed? I'm thinking about cloud interoperability, hybrid cloud, so that you could run on-premises and then have those applications seamlessly move to a public cloud environment and back.

Andrew, starting with you, what it is about the UNIX model and The Open Group certification and standardization model that we might apply to such efforts as OpenStack, or Cloud Foundry, or some other efforts to make a seamless environment for the hybrid cloud?

Exciting problem

Josey: In our standards process, we're very much able to take on almost any problem, and this certainly would be a very exciting problem for us to tackle to bring parties together. We're able to bring different parties together, looking for commonality to try and build the consensus.

We get people in the room to talk through the different points of view. What The Open Group is able to do is to provide a safe harbor where the different vendors can come in and not be seen as talking in an anti-competitive position, but actually discussing the differences and their implementations and deciding what’s the best common way to go forward who is setting a standard.

Gardner: Anyone else on the relationship between UNIX and hybrid cloud in the next several years?

Johnson: I can talk to it a little bit. The real opportunity, and I hope people reading this, and especially the OpenStack community listens, is that true innovation can be best done on a foundation. In OpenStack, it’s a number of communities that are loosely affiliated delivering great progress, but there is interoperability, and it’s not with intent, but it’s just people are moving fast. If some foundation elements can be built, that's great for them because then we, as vendors, can more easily support the solutions that these communities are bringing to us, and then we can deliver to our customers.

In hybrid cloud environments, what UNIX brings to customers is security, reliability, and flexibility.

Cloud computing is the Wild West. We have Azure, OpenStack, AWS, and could benefit from some consistency. Now I know that each of our companies will go to great lengths to make sure that our customers don't see that inconsistency. So we bear the burden for that, but what if we could spend more time helping the communities be more successful rather than, as I mentioned before, reinventing the wheel? There is a real opportunity to have that synergy.

Kyle: In hybrid cloud environments, what UNIX brings to customers is security, reliability, and flexibility. So the Wild West comment is very true, but UNIX can present that secure, reliable foundation to a hybrid cloud environment for customers.

Gardner: Let’s look at this not just through the lens of technology but some of the more intangible human cultural issues like trust. It seems to me that, at the end of the day, what would make something successful as long as UNIX has been successful is if enough people from different ecosystems, from different vantage points, have enough trust in that process, in that technology. And through the mutual interdependency of the people in that ecosystem they keep it moving forward. So let’s look at this from the issue of trust and why we think that that's going to enable a long history for UNIX to continue.

Josey: We like to think The Open Group is a trusted party for building standards and that we hold the specification in trust for the industry and do the best thing for it. We're fully committed always to continue working in that area. We're basically the secretariat, and so we're enabling our customers to save a lot of cost. We're able to divide up the cost. If The Open Group does something once, that’s much cheaper than everybody doing the same thing themselves.

Gardner: Darrin, do you agree with my premise that trust has been an important ingredient that has allowed UNIX to be so successful? How do we keep that going?

One word: Open

Johnson: The foundation of UNIX, even going back to the original development, but certainly since standards came about is the one word “open.” You can have an open dialogue to which anybody is invited. In the case of the Austin Group, it’s everybody. In the case of any of the efforts around UNIX, it’s an open process, it’s open involvement, and in the case of The Open Group, which is kind of another open, it’s vendor-neutral. Their goal is to find a vendor-neutral solution.

Also look at this way. We have IBM, HPE, and Oracle sitting here, and I’ll say virtually Linux. Other communities that are participating are coming to mutual agreements, and this is what we believe is best.

And you know what, it’s open to disagreement. We disagree all the time, but in the end what we deliver and execute is of mutual agreement, so it’s open, it’s deterministic, and we all agree on it.

We disagree all the time, but in the end what we deliver and execute is of mutual agreement, so it’s open, it’s deterministic, and we all agree on it.

If I were a customer, IT professional, or even a developer, I'd be going, "This foundation is something on which I want to innovate, because I can trust that it will be consistent." The Open Group is not going to go away any time soon, celebrating 20 years of supporting the standard. There's going to be another 20 years.

And the great thing is that there is lot of opportunity to innovate in computer science in general, but the standard is building that foundation, taking advantage of topics like security, virtualization, mobility, and the list goes on. We even have opportunity to in a open way build something that people can trust.

Gardner: Tom, openness and trust, a good model for the next 20 years?

Mathews: It is a good model. Darrin touched on it. If we need proof of it, we have 20 years in proof of it. The Open Group has brought together major competitors and, as Darrin said, it’s always been very open, and people have always -- even with disagreement -- come to a common consensus around stuff. So The Open Group has been very effective establishing that kind of environment, that kind of trust.

Listen to the podcast. Find it on iTunes. Get the mobile app. Download the transcript. Sponsor: The Open Group.

You may also be interested in:

• The Open Group president, Steve Nunn, on the inaugural TOGAF User Group and new role of EA in business transformation
•  A Tale of Two IT Departments, or How Cloud Governance is Essential in the Bimodal IT Era
• Securing Business Operations and Critical Infrastructure: Trusted Technology, Procurement Paradigms, Cyber Insurance 
• Enterprise Architecture Leader John Zachman on Understanding and Leveraging Synergies Among the Major EA Frameworks
• Cybersecurity standards: The Open Group explores security and safer supply chains
• Explore synergies among major Enterprise Architecture frameworks with The Open Group
• Health Data Deluge Requires Secure Information Flow Via Standards, Says the Open Group's New Healthcare Director
• The Open Group Amsterdam Conference Panel Delves into How to Best Gain Business Value from Open Platform 3.0
• Healthcare Among Thorniest and Yet Most Opportunistic Use Cases for Boundaryless Information Flow Improvement
• Gaining Dependability Across All Business Activities Requires Standard of Standards to Tame Dynamic Complexity, Says The Open Group CEO
• Big Data success depends on better risk management practices like FAIR, say conference panelists

Digital payments now catching on across the business world too

The next BriefingsDirect business innovation thought leadership discussion focuses on how digital payments are catching on for many more companies in the business world following the popularity of services like Apple Pay in the consumer world.

We'll now explore how digital payment solutions are changing the game for small companies like 487 Consulting Services, which is seeing faster and simpler payments using AribaPay. And we will hear more about how B2B digital payment services like AribaPay are expanding around the globe.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy.

To learn more about the future of business payments innovation, we're joined by Drew Hofler, Senior Director of Marketing at SAP Ariba; Ken Crouse, Principal Consultant and Owner at 487 Consulting Services in Folsom, California, and Bill Dulin, Vice President of Commercial Payments at Discover in Chicago. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: Drew, for almost anything that consumers want to buy these days there's a swipe or a card chip, and we are now into wireless connectivity for payments. And yet, with business-to-business (B2B), we're still many times faxing and writing paper checks -- and it's largely still a manual process.

So why such a dichotomy between what people can do as a consumer buying gasoline, for example, and a company buying critical goods and services?

Hofler: It's fundamentally the difference between payments in B2B and the consumer world. For consumers, it's relatively simple.

Everything that you're going to buy is in a single cart at the time of payment, and it all takes place in one spot. The information and the payment itself happen together.

In the B2B world, that is just simply not the case. In the B2B world, you have an invoice that comes in for a good delivered or service rendered, and then payment may happen 30, 45, 60, 90 days later, and that payment may include more than one invoice.

Oftentimes in the B2B context, it includes hundreds of invoices on a single credit of funds into an account. So there's a huge gap between the payment and the information, and that’s what we're trying to solve. That's where the innovation needs to come, bringing that information that’s necessary for all parties to know what's being paid for, when, and why, bringing that together with the settlement of funds in a very secure environment.

Closing the gap

Gardner: But we are closing the gap. Tell us a little bit about AribaPay. How long has it been around and why is it now in a position to begin closing that gap even more rapidly than ever?

Hofler: We launched general availability of AribaPay a little over year ago, and we started here in North America. We've seen rapid growth and we just announced that we're expanding into Canada with our partner, Discover. We are also expanding, later in the year, into Europe and Latin America.

 Hofler

Even though the payment systems are different, the fundamental issue with B2B payments -- the disconnect between information and the settlement of funds -- is the same no matter where you go geographically.

So that's why we're taking it global, and why we're in a position to really change the game and innovate in B2B payments. We sit at the nexus of the digital network age, which is a very different age from where payments began.

You have electronic payments like ACH in the US (or SEPA in Europe) and these types of electronic payment schemes were created back in the '70s, based on a paradigm of that time, which was COBOL-based mainframes behind brick walls, and there was no way to connect a buyer's systems with the supplier's systems.

But now, we live in the digital age, where the Ariba network connects millions of buyers and suppliers together to transact and move terabytes of data in real time between back-end systems.

Instead of doing what B2B payments and electronic payments have done in the past, which is try to take a small subset of that information out and attach it to the payment, (using the ACH or to the SEPA formats, 140 characters in Europe, which is the same as a tweet, or 80 usable characters in the US) we're taking the payment and attaching it to all of this information that’s already on the Ariba Network, the purchase order (PO), the invoice, the reason why the invoice maybe paid a little less than was expected. All of that information is fully available on the network.

We make it visible with the payment, so that both buyers and suppliers know exactly what's being paid, why it's being paid, what this million-dollar deposit is, even if it's a thousand invoices, and why it may be a little different than the supplier was expecting. All of that is fully visible and available on the Ariba Network.

Gardner: Bill, tell us a bit about the role that Discover plays in all this. And how do you feel about the gap closing between what happens in the consumer space and what can now happen in the business space?

Facilitating payments

Dulin: I think I would like to start off with what AribaPay is not, and it's not a card offering. Usually, when people see the Discover logo, they're thinking of a credit-card offering, but this is not that. We're using our infrastructure to facilitate commercial payments.

Dulin

In that case, we’re making sure that we're gathering the bank account information, we're acting as the financial institute of record, we're boarding the suppliers, so all of that information is now in our trusted network. That's how we show up as the financial institution, as the bank. We then move the money and, as Drew talked about a little bit earlier, along with that data as well. That's really where the gap is closing. We're bringing the data and the financial transaction together.

Gardner: Drew, this is not just for large companies. It should be for any company. The long tail, if you will, the larger number of people involved, will be those small-to-medium size businesses (SMBs). Is there something in it that's different or special for them other than your Global 2000 corporations?

Hofler: It’s particularly different for the receivers of payments on that long tail. The large companies have the IT resources they need to manage the complex electronic payments that are available today. That's based on EDI and things like that, and that's great.

The midsize to the smaller suppliers simply don't have the technical resources to consume the information in those formats. They just can't do it. What AribaPay really does is it makes it as simple as possible.

But then the midsize to the smaller suppliers simply don't have the technical resources to consume the information in those formats. They just can't do it. What AribaPay really does is it makes it as simple as possible.

It is as simple as an email with the information about the payment and a link into their account in the Ariba Network that they can visibly see all the information around their payment in a very nice UI. For example, if they were expecting a $1,000 payment and they got $900, the big question is why. There may be 10 invoices on that payment.

They come in, click that link, and come right into their account on the network. They see the payment ID for that $900 that they have, and we show them exactly what was invoiced, the $1,000. You expected $1,000, but you received $900, and here exactly is where the difference is from.

They have hyperlinks to go into the invoice. They can see the comments that may have been made on how maybe something was broken on the pallet, and so they only paid for 9 items instead of 10.

All of that is a very simple online experience.

Gardner: Ken, tell me a bit about 487 Consulting Services, what you do, and then we'll ask about how you like to get paid?

One-man shop

Crouse: 487 Consulting Services is my personal business. It's a one-man shop. I literally get up in the morning, walk over and turn on the coffee pot and walk over to my desk. That's probably the best part of being an independent.

Crouse

The other side of being an independent, though, is that I'm responsible for every single aspect of the business from submitting the financial filings that we did with Discover and getting on board with everybody and actually doing the work for which I'm getting paid. It's all done by me and is controlled by me.

There is no IT department. There is no human resources department. There is no large infrastructure behind me -- it's just me. I came to SAP Ariba via a customer that said they wanted to pay me that way.

Initially, I was a little apprehensive because I was expecting that I'd have to learn a new program. I could just flash back to COBOL in college back in the '80s, and that was petrifying, but the simplicity and the transparency of SAP Ariba was just refreshing.

The first webinar I attended, although scheduled for one hour, only lasted about 30 minutes because of the simplicity and then, within a couple of days, I was able to get all my paperwork together for Discover, and I was live on Ariba within less than a week.

Now, with the Ariba Network, when it comes time to do my invoice and do it about twice a month, I open my Ariba account, identify the purchase order to be billed, click the service that's to be billed and click the submit button.

Two weeks later, I received my first series of payments through Ariba and have been now receiving payments since the first of January 2015. Ariba has processed something north of 300 invoices for me amounting to probably 500 to 600 individual tasks.

Gardner: I think there are going to be more and more folks like you, smaller businesses, independents working to provide discrete services throughout our economy, around the world, many of them working off just the smart phone.

So this is an important part of our growing economy, but also it’s important for an organization like yours to have great visibility to know when the money is coming and when to expect it. Cash flow is pretty important.

So tell me a little bit about that visibility and expectation, and how this system worked better than paper, faxes, and checks?

Previous system

Crouse: It's probably best that I just take a step back from that and review where I was before Ariba, and like you mentioned, it was a paper invoicing system. My customer required that each purchase order be on a separate piece of paper for the purposes of invoicing.

So I might create 15 or 20 invoices, put them all in the same envelope with a nice little transmittal sheet, mail them off. Then, 75 days later, when I'm not getting paid for some invoice, I would then get hold of them, and they would say "Oops, your invoice isn't in our system. And I'd start all over again. That would be a time out from work. I had to stop what I was doing, resubmit the invoice, and then start the clock all over again.

Now, with the Ariba Network, when it comes time to do my invoice and do it about twice a month, I open my Ariba account, identify the purchase order to be billed, click the service that's to be billed and click the submit button. Quite literally, the invoicing is just that simple.

Within a matter of minutes, I receive recognition that the invoice is in the system, as opposed to waiting 75 days for confirmation that it's not there. I receive a positive affirmation within just a matter of minutes.

And then, within 48 to 72 hours, I have a customer who has acknowledged and has approved that invoice for payment. At that point, I know with certainty that that payment is going to come in and on a date certain. I can forecast my cash accordingly and then go on vacation. I don't have to worry about it.

When I get the notifications of the payment being in there, it's broken down line item by line item that corresponds to the exact tasks that I have done for that particular payment. I enjoy the fact that it is all in one payment and broken out that way.

Gardner: Also, Drew mentioned this opportunity for more rich information to be associated with the transaction, remittance information for example. Have you been able to avail yourself of that and is that an important part of what you're doing, being able to see all the information associated with an invoice or a payment process?

Crouse: When I get the notifications of the payment being in there, it's broken down line item by line item that corresponds to the exact tasks that I have done for that particular payment. I enjoy the fact that it is all in one payment and broken out that way.

In the past, a year and a half ago, I might receive individual payments for all of those invoices. I'd get an envelope in the mail that might have a dozen checks in it and then, I'd have to go back and reconcile one check against one invoice. It was just a very time consuming and very clumsy effort.

The other part is that I wouldn’t necessarily get paid for all of my invoices submitted on a given date at the same time. I'd get paid for 10 of the 12 invoices and then would have to start this tail-wagging-the-dog episode of chasing around payments on the other invoices and payments. Although the majority of them might be paid in 60 days, it wasn't uncommon that they would stretch out to 120 or 150 days.

Digitizing processes

Gardner: Bill, any thoughts from the Discover perspective on the ability to not just repave cow paths, but actually do things in business that could not have been done before, given that we are digitizing these processes?

Dulin: A key for us in this, and what we haven’t talked about too much, is the compliance that’s around it. So as we are moving these payments, knowing who the customer is, anti-money laundering, all the regulatory compliance that goes around it. That makes it a more robust payment.

We become more sophisticated as the technology wraps around that payment, to know where it's going, where it should be going. If something has happened that triggers it -- it makes us stop and take a look, to make sure. Sometimes, we talk about purposeful friction. Something triggered an event that made us stop the payment and take a look around and make sure that we have it.

From our perspective in this case, it's not so much of the technology; it’s pulling that sensitive information out of enterprise resource planning (ERP) programs or other places that it shouldn't be and then putting it in a financial institution, again, using that technology around it to help secure that.

Gardner: Now, we heard a lot at the recent Ariba Live 2016 Conference about risk reduction and visibility in the supply chain, that it's really about managing your supply chain. Is there something about using AribaPay, when you have all that data associated that gives people more insight into their supply chain than they may have had, auditability, the ability to further define what it is that they want in terms of best practices, Drew?

More data is better than less data, as long as you can consume it and put it in a usable format, and that's really what we are doing.

Hofler: More data is better than less data, as long as you can consume it and put it in a usable format, and that's really what we are doing.

Knowing exactly who is being paid and removing the opportunities for fraud in the payment process is huge, and AribaPay really removes those opportunities for fraud or a vast majority of them.

We have this whole platform of information and data about the interactions between a buyer and their supplier, from the moment that they source, to when they procure, to the PO, to the invoice, to the payment going through. They can see the on-time performance and they can see how often that supplier requests early payment, if they're using Dynamic Discounting on the Ariba Network, and they can feed that back into the procurement side and start to define payment terms as a result of that at the very beginning.

Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy. Sponsor: SAP Ariba.

You may also be interested in:

• How new technology trends disrupt the very nature of business
• SAP Ariba Chief Strategy Officer on the Digitization of Business and the Future of Technology
• How New Technology Trends Disrupt the Very Nature of Business
• Procurement in 2016 -- The Supply Chain Goes Global
• Is 2016 the Year that Accounts Payable Becomes Strategic?
• Winning the B2B Commerce Game: What Sales Organizations Should Do Differently
• Can Great Design Really Impact Global Commerce?
• Ariba’s digital handshake helps Caesars up the ante on supply chain diversity
• Detecting and Eradicating Slavery and Other Labor Risks Across Global Supply Chains through Business Networks
• Ariba elevates business user experience with improved mobile apps and interfaces
• Ariba's product roadmap for 2015 leads to improved business cloud services
• Networks: The new model for B2B business, a panel discussion