The next
BriefingsDirect Voice of the Customer digital business transformation case study highlights how
Independent Health in Buffalo, New York has entered into a next phase of "strategic
DevOps."
After a two-year drive to improve software development, speed to value,
and improved user experience of customer service applications,
Independent Health has further extended advanced testing benefits to ongoing apps
production and ongoing performance monitoring.
Learn here how
the reuse of proven performance scripts and replaying of synthetic
transactions that mimic user experience have cut costs and gained early
warning and trending insights into app behaviors and system status.
Here to describe how to attain such new strategic levels of DevOps benefits are
Chris Trimper, Manager of Quality Assurance Engineering at Independent Health in Buffalo, New York, and
Todd DeCapua,
Senior Director of Technology and Product Innovation at CSC Digital
Brand Services Division and former Chief Technology Evangelist at
Hewlett Packard Enterprise (HPE). The discussion is moderated by
BriefingsDirect's Dana Gardner, Principal Analyst at
Interarbor Solutions.
Here are some excerpts:
Gardner:
What were the major drivers that led you to
increase the way in which you use DevOps, particularly when you're
looking at user-experience issues in the field and in production?
Trimper:
We were really hoping to get a better understanding of our users and
their experiences. The way I always describe it to folks is that we
wanted to have that opportunity to almost look over their shoulder and
understand how the system was performing for them.
Whether
your user is internal or external, if they don't have that good user
experience, they're going to be very frustrated and they're going to
have a poor experience. Internally, time is money. So, if it takes
longer for things to happen, and you get frustrated potential turnover,
it's an unfortunate barrier.
Gardner: What kind
of applications are we talking about? Is this across the spectrum of
different type of apps, or did you focus on one particular type of app
to start out?
End users important
Trimper:
Well, when we started, we knew that the end user, our members, were the
most important thing to us, and we started off with the applications
that our servicing center used, specifically our
customer relationship management (CRM) tool.
If the member information doesn’t pop fast when a
member calls, it can lead to poor call quality, queuing up calls, and it
just slows down the whole business. We pride ourselves on our
commitment to our members. That goes even as far as, when you call up,
making sure that the person on the other end of the phone can service
you well. Unfortunately, they can only service you as well as the data
that’s provided to them to understand the member and their benefits.
Gardner:
It’s one thing to look at user experience through performance, but it's
a whole new dimension or additional dimension when you're looking at
user experience in terms of how they utilize that application, how well
it suits their particular work progress, or the processes for their
business, their line of business. Are you able to take that additional
step, or are you at the point where the feedback is about how users
behave and react in a business setting in addition to just how the
application performs?
Trimper: We're starting to
get to that point. Before, we only had as much information as we were
provided about how an application was used or what they were doing.
Obviously, you can't stand there and watch what they're doing 24x7.
Lately,
we've been consuming an immense amount of log data from our systems and
understanding what they're doing, so that we can understand their
problems and their woes, or make sure that what we're testing, whether
it's in production monitoring or pre-production testing, is an accurate
representation of our user. Again, whether it’s internal or external,
they're both just as valuable to us.
Gardner:
Before we go any further, Chris, tell us a little bit about Independent
Health. What kind of organization is it, how big is it, and what sort
of services do you provide in your communities?
Trimper:
We're a healthcare company for the Western New York area. We're a
smaller organization. We define the red-shirt treatment that stands for
the best quality care that we can provide our members. We try to be very
proactive in everything that we do for our members as well. We drive
members to the provider to do preventative things, that healthier
lifestyle that everybody is trying to go for.
Gardner: Todd, we're hearing this interesting progression toward a
feedback loop
of moving beyond
performance monitoring into behaviors and use patterns
and improving that user experience. How common is that, or is
Independent Health on the bleeding edge?
Ahead of the curve
DeCapua:
Independent Health is definitely moving with, or maybe a little bit
ahead of, the curve in the way that they're leveraging some of these
capabilities.
If we were to step back and look at where we've been
from an industry perspective across many different markets,
Agile
was
hot, and now, as you start to use Agile and break all the right internal
systems for all the right reasons, you have to start adopting some of
these DevOps practices.
Independent
Health is moving a little bit ahead on some of those pieces, and
they're probably focusing on a lot of the right things, when you look
across other customers I work with. It's things like speed of time to
value. That goes across technology teams, business teams, and they're
really focused on their end customer, because they're talking about
getting these new feature functions to benefit their end customers for
all the right reasons.
You heard Chris talking about
that improved end-user experience about around their customer service
applications. This is when people are calling in, and you're using tools
to see what’s going on and what your end users are doing.
There's
another organization that actually recorded what their customers were
doing when they were having issues. That was a production-monitoring
type thing, but now you're recording a video of this. If you called
within 10 minutes of having that online issue, as you are calling in and
speaking with that customer service representative, they're able to
watch the video and see exactly what you did to get that error online to
cause that phone call. So having these different types of users’
exceptions, being able to do the type of production monitoring that
Independent Health is doing is fantastic.
I do think that Independent Health is hitting the bleeding edge on that piece. That’s what I've observed.
Another area that Chris was telling me about is some of the
social media
aspects and being able to monitor that is another way of getting
feedback. Now, I do think that Independent Health is hitting the
bleeding edge on that piece. That’s what I've observed.
Gardner:
Let’s hear some more about that social media aspect, getting additional
input, additional data through all the available channels that you can.
Trimper:
It would be foolish not to pay attention to all aspects of our members,
and we're very careful to make sure that they're getting that quality
that we try to aim for. Whether it happens to be
Facebook,
Twitter, or some other mechanism that they give us feedback on, we take all that feedback very seriously.
I
remember an instance or two where there might have been some negative
feedback. That went right to the product-management team to try to
figure out how to make that person’s experience better. It’s
interesting, from a healthcare perspective, thinking about that.
Normally, you think about a member’s copay or their experience in the
hospital. Now, it's their experience with this application or this web
app, but those are all just as important to us.
Broadened out?
Gardner: You started this with those customer-care applications. Has this
broadened out into other application development? How do you plan to
take the benefits that you've enjoyed early and extend them into more
and more aspects of your overall IT organization?
Trimper:
We started off with the customer service applications and we've grown
it into observing our provider portals as well. A provider can come in
and look at the benefits of a member, the member portal that the members
actually log in to. So, we're actually doing production monitoring of
pretty much all of our key areas.
We also do
pre-production monitoring of it. So, as we are doing a release, we don’t
have to wait until it gets to production to understand how it went.
We're going a little bit beyond normal performance testing. We're running
the same exact types of continuous monitoring in both our
pre-production region and our production regions to ensure that quality
that we love to provide.
Gardner: And how are
the operations people taking this? Has this been building bridges? Has
this been something that struck them as a foreign entity in their
domain? How has that gone?
Trimper: At first, it
was a little interesting. It felt like to them it was just another
thing that they had to check out and had to look at, but I took a unique
approach with it. I sat down and talked to them personally and said,
"You hear about all these problems that people have, and it’s impossible
for you to be an expert on all these applications and understand how it
works. Luckily, coming from the quality organization, we test them all
the time and we know the business processes."
The
way I sold it to them is, when you see an alert, when you look at the
statistics, it’s for these key business processes that you hear about,
but you may not necessarily want to know all the details about them or
have the time to do that. So, we really gave them insight into the
applications.
As far as the alerting, there was a
little bit of an adoption practice for that, but overall we've noticed a
decrease in the number of support tickets for applications, because
we're allowing them to be more proactive, whether it’s proactive of an
unfortunately blown
service-level agreement (SLA), or it’s a degradation in quality of the performance. We can observe both of those, and then they can react appropriately.
Gardner: Todd, he actually sat down and talked to the production people. Is this something novel? Are we seeing more of that these days?
DeCapua:
We're definitely seeing more of it, and I know it’s not unique for
Chris. I know there was some push back at the beginning from the
operations teams.
There was another thing that was
interesting. I was waiting for Chris to hit on it, and maybe he can go
into it a little bit more. It was the way that he rolled this out. When
you're bringing a monitoring solution in, it’s often the ops team that’s
bringing in this solution.
Making it visible
What’s
changing now is that you have these application-development testing
teams that are saying, "We also want to be able to get access to these
types of monitoring, so that our teams can see it and we can improve
what we are doing and improve the quality of what we deliver to you, the
ops teams. We are going to do instrumenting and everything else that we
want to get this type of detail to make it visible."
Chris
was sharing with me how he made this available first to the directors,
and not just one group of directors, but all the directors, making this
very plain-sight visible, and helping to drive some of the support for
the change that needed to happen across the entire organization.
As
we think about that as a proven practice, maybe Chris is one of the
people blazing the trail there. It was a big way of improving and
helping to illuminate for all parties, this is what’s happening, and
again, we want to work to deliver better quality.
Gardner: Anything to add to that, Chris?
Trimper:
There were several folks in the development area that weren’t
necessarily the happiest when they learned that the perception of what
they originally thought was there and what was really there in terms of
performance wasn’t that great.
It was a big way of improving and helping to illuminate for all parties, this is what’s happening.
One
of the directors shared an experience with me. He would go into our
utilities and look at the dashboards before he was heading to a meeting
in our customer service center. He would understand what kind of looks
he was going to be given when he walked in, because he was directly
responsible for the functionality and performance of all this stuff.
He
was pleased that, as they went through different releases and were able
to continually make things better, he started seeing everything is
green, everything is great today. So, when I walk in, it’s going to be
sunshine and happiness, and it was sunshine and happiness, as opposed to
potentially a little bit doomy and gloomy. It's been a really great
experience for everyone to have. There's a little bit of pain going
through it, but eventually, it has been seen as a very positive thing.
Gardner:
What about the tools that you have in place? What allows you to provide
these organizational and cultural benefits? It seems to me that you
need to have data in your hands. You need to have some
ability to execute once you have got that data. What’s the technology
side of this; we've heard quite a bit about the people and the process?
Trimper:
This whole thing came about because our CIO came to me and said. "We
need to know more about our production systems. I know that your team is
doing all the performance testing in pre-production. Some of the folks
at HPE told me about this new tool called
Performance Anywhere. Here it is, check it out, and get back to me. "
We
were doing all the pre-production testing and we learned that all the
scripts that we did, which had already been tried and true and been
running and continuously get updates as we get new releases, could just
be turned into these production monitors. Then, we found through using
the tool, through our trial, and now all of our two plus years that we
have been working with it that it was a fairly easy process.
Difficult point
The
most difficult point was understanding how to get production data that
we could work with, but you could literally take a test on your
VUGen script and turn it into a production monitor in 5-10 minutes, and that was pretty invaluable to us.
That
means that every time we get a release, we don’t have to modify two
sets of scripts and we don’t have two different teams working on
everything. We have one team that is involved in the full life cycle of
these releases and that can very knowledgeably make the change to those
production monitors.
Gardner: HPE Performance
Anywhere. Todd, are lot of people using it in the same fashion where
they're getting this dual benefit from pre-production and also in
deployment and operations?
DeCapua: Yes, it’s
definitely something that’s becoming more-and-more aware. It’s a
capability that's been around for a little while. You'll also hear about
things like
IT4IT,
but I don’t want to open up that whole can of worms unless we want to
dive into it. But as that starts to happen, people like Chris, people
like his CIO, want to be able to get better visibility into all systems
that are in production, and is there an easy way to do that? Being able
to provide that easy way for all of your stakeholders and all of your
customers are capabilities that we're definitely seeing people adopt. It
was a big way of improving and helping to illuminate for all parties,
this is what’s happening
That
means that every time we get a release, we don’t have to modify two
sets of scripts and we don’t have two different teams working on
everything.
Gardner: Can you provide a bit more detail in terms of the actual products and services that made this possible for you, Chris?
Trimper: We started with our
HPE LoadRunner scripts, specifically the VUGen scripts, that we were able to turn into the production monitors. Using the
AppPulse Active tool from the AppPulse suite of tools, we were able to build our scripts using their
SaaS infrastructure and have these monitors built for us and available to test our systems.
Gardner:
So what do you see in our call center? Are you able to analyze in any
way and say, "We can point to these improvements, these benefits, from
the ability for us to tie the loop back on production and quality
assurance across the production spectrum?"
Trimper:
We can do a lot of trend analysis. To be perfectly honest, we didn’t
think that the report would run, but we did a year-to-date trend
analysis and it actually was able to compile all of our statistics. We
saw really two neat things.
When you had open
enrollment, we saw this little spike that shot up there, which we would
expect to see, but hopefully we can be more prepared for it as time
goes. But we saw a gradual decrease, and I think, due to the ability to
monitor, due to the ability to react and plan better for a better
performing system, through the course of the year, for this one key
piece of pulling member data, we went from an average of about 12-14
seconds down to 4 seconds, and that trend actually is continuing to go
down.
I don’t know if it’s now 3 or less today, but if
you think about that 12 or 14 down to about 4, that was a really big
improvement, and it spoke volumes to our capabilities of really
understanding that whole picture and being able to see all of that in
one place was really helpful to us.
Where next?
Gardner:
Looking to the future, now that you've made feedback loops demonstrate
important business benefits and even move into a performance benefit for
the business at large, where can you go next? Perhaps you're looking at
security and privacy issues, given that you're dealing with compliance
and regulatory requirements like most other healthcare organizations.
Can you start to employ these methods and these tools to improve other
aspects of your SLAs?
Trimper: Definitely, in
terms of the SLAs and making sure that we're keeping everything alive
and well. As for some of the security aspects, those are still things
where we haven’t necessarily gone down the channels yet. But we've
started to realize that there are an awful lot of places where we can
either tie back or really start closing the gaps in our understanding of
just all that is our systems.
Gardner: Todd,
last word, what should people be thinking about when they look at their
tooling for quality assurance and extending those benefits into full
production and maybe doing some cultural bonding at the same time?
The culture is a huge piece. No matter what we talk about nowadays, it starts with that.
DeCapua:
The culture is a huge piece. No matter what we talk about nowadays, it
starts with that. When I look at somebody like Independent Health, the
focus of that culture and the organization is on their end user, on
their customer.
When you look at what Chris and his
team has been able to do, at a minimum, it’s reducing the number of
production incidents. And while you're reducing production incidents,
you're doing a number of things. There are actually hard costs there
that you're saving. There are opportunity costs now that you can have
these resources working on other things to benefit that end customer.
We've
talked a lot about DevOps, we've talked a lot about monitoring, we've
mentioned now culture, but where is that focus for your organization?
How is it that you can start small and incrementally show that value?
Because now, what you're going to do is be able to illustrate that in
maybe two or three slides, two or three pages.
But
some of the things that Chris has been doing, and other organizations
are also doing, is showing, "We did this, we made this investment, this
is the return we got, and here's the value." For Independent Health,
their customers have a choice, and if you're able to move their
experience from 12-14 seconds to 4 seconds, that’s going to help. That’s
going to be something that Independent Health wants to be able to share
with their potential new customers.
As far as
acquiring new customers and retaining their existing customers, this is
the real value. That's probably my ending point. It's a culture, there
are tools that are involved, but what is the value to the organization
around that culture and how is it that you can then take that and use
that to gain further support as you move forward?
Listen to the podcast. Find it on iTunes. Get the mobile app. Read a full transcript or download a copy. Sponsor: Hewlett Packard Enterprise.
You may also be interested in:
