Changing Your Ticketing System

There's always room for improvement.

If you've been with Acquia for a while you'll have noticed that we recently switched ticketing systems. You probably also had a litany of things that you really disliked about our old system, starting with the iframe and ending with…well, there probably was no end.

We were pretty tired of it too. It had worked well enough when we were a smaller company; though we all knew we could have built something better internally, as a growing company we had to make tough choices as to where we put our efforts.

In December of 2013 we started a looking in earnest for a new ticketing system. We worked our way through a very detailed vendor selection process (including DIY) and ultimately made our choice: Zendesk. We picked them for a lot of reasons, but both the customer service team and the engineering team felt that they could satisfy our core needs while providing the flexibility to build out additional functionality. From an engineering standpoint we were swayed by the fact that their own front-end was built on their API. We knew that the API would always be a first-class citizen with Zendesk and not an afterthought.

From there the planning began in earnest. We took all the hopes and dreams of our customers, support reps, and UX team's research; and constructed a detailed requirements plan from them. I can tell you that our backlog was littered with hundreds of stories either describing pain that needed to be addressed, or describing what a better ticketing system would look like.

You need it done by when?!

Then plans changed: It was contract renewal time with the old ticketing system and the decision was made that we didn't want to renew for the term we were offered.

It was January 27th; the contract expired on February 15th. We had 20 days to move from the old system to Zendesk. Just to make it a little bit more fun, we set our go-live date as February 13th because who wants to do a major release on a Saturday?

Anyone who has been in software development for some time has hit these types of events. Sometimes they fail horribly, sometimes you manage to just barely pull it off, and sometimes they succeed wildly. I'd like to think that our effort was a wild success and I'll tell you why.

The first thing that gave us a leg up is something we called the Ticketpool. Our old ticketing system had a lot of shortcomings. It's hard to pinpoint the worst, but one of them was the API. It was clunky, could sometimes involve 20-40 calls to accomplish a single task and was easy to hit rate limiting. Another was the reportingit required a PhD in the arcane arts to get any data out. As a result we built the Ticketpool site on Drupal and kept a steady export of tickets flowing into it so that any complex reporting and data munging could happen there. This also gave us an ideal platform from which to run a migration.

The second thing that provided a jumpstart was a system called Suponcall. Again, due to shortcomings in the previous ticketing system we had a Drupal site which integrated with a calendar system, watched tickets, and would notify and escalate to people when tickets came in that needed attention. As it turns out, no ticketing system that met our other requirements really did calendaring and email/sms/phone escalation well, so we were lucky that we were already prepared to handle this functionality ourselves.

The final thing that gave us the confidence to say, we can do this, was that our lead developer on the project, George Cassie, had already started building a Drupal front-end on top of Zendesk's API which we could directly integrate into the Acquia Network rather than an iframe or a vendor provided portal.

With these three things in place, and after consulting with UX, we decided that yes, we can meet the deadline and gave the nod from our perspective.

Work immediately got underway to pare down the feature list. In the words of Lone Star, take only what you need to survive. This process was done in two days. The UX team then worked fast and furiously to take some existing designs and mesh them with the altered feature set, then get them into the hands of the devs while simultaneously testing prototypes with a group of customers.

The devs started shoring up the initial prototype UI and worked hand-in-hand with the designers in a highly agile fashion. At the same time old tickets were being cleaned up in the ticketing system that was on the way out and migration scripts were being written and tested.

A final fourth thing which factored into our decision to say ‘go' (I know…I said there were only three) was that the switch-over would occur during Acquia Build 2014. Our Build Weeks are a time when we get all of engineering together in our Burlington, MA offices and groups get together to build cool new things and try out new ideas.

Having the sprint wrap up during Build Week meant that the entire team working to build the new UI and do the migration would be together in one room. Without this, the timeframe would have become much more difficult. The engineers were also all very gracious in giving up the chance to build shiny new things and work 12-16 hour days to complete the work.

Iterate, then iterate some more.

Overall, the iterative process between the designers, engineers, and the support group was something we will hold up as an ideal example for some time.

The informal daily and sometimes hourly reviews between all the parties ensured that no one was out of the loop and no one wandered too far afield. You do get this sort of daily sync with scrum, but sometimes scrum reports can degenerate into, looked at tickets, which isn't terribly useful. Everyone feeling the crunch ensured that people shared exactly what they were doing because they needed code reviews and UX reviews to move onto the days' work.

I can't even begin to describe the amount of work that happened--most of it in about twelve days. From the customer-facing UI work, to email templates, workflows and triggers, to time tracking integrations and SLA management processes. Just listing the stories and brief descriptions of each would take more room than this entire post. Suffice to say that a lot of people in Engineering and Support worked their tails off.

The migrations proved to be the hardest part; though this wasn't unexpected. In the end a group of us jammed on them for 3 days straight to get the scripts right. The way we did it was to bulk push an initial 100,000 or so tickets into Zendesk from Ticketpool and then periodically sync new tickets and changes into Zendesk. This turned out to be very useful because the previous ticking system's export functionality was broken so we wouldn't have been able to use their bulk exporter anyway.

In the end all dev work and testing was completely wrapped up about six hours before go-live. Some beers were shared and a few card games were played as we counted down the hours to the switch flipping. Making this process easier was our internal concept of ‘feature flags'. This is a system we use in the Network/Cloud UI where we wrap functionality in a permission we can apply to a role, or to a flag that can be set on a subscription. This allows us to release the code for a new feature, enable it for select individuals and then add the permission to a role assigned to all users when we want it to go live. It also makes it easy to roll back if there is an issue.

So when 10PM on February 13th rolled around all we had to do was add a permission to our ‘subscriber' role and the system was live. Surrounding that was a lot of process we put in place to make sure that notices were displayed, inbound and outbound email processing switched over, etc… but by and large it was easily accomplished in the hour we gave ourselves and was completely successful.

What does the future hold?

In the weeks following we have received a wealth of great feedback, most of it overwhelmingly positive and constructive. We've made a few tweaks based on what customers have said and have some more updates coming.

Because moving to Zendesk allowed us to essentially build our own front-end on their ticketing system the sky is now the limit for how we present tickets and provide feedback/information to our customers on their support requests. Expect some exciting changes as we bring more data to you where you are in our UI or in your own tools.

What's even better is that our initial roadmap had us working on this migration well into Q3; but because of the accelerated timeframe we had to ask whether or not we really needed every piece of functionality. It turned out that a lot of the things we really thought we needed were workarounds we had developed in the old ticketing system and we were better for not bringing them forward.

As a result we will probably wrap up the entire feature roadmap we had planned for 2014 around mid-summer.

Overall it was a wild success… not that we want to make 20 day system transitions a habit :)