Scrum, Drupal & Content Hub (Pt 2)

In Part 1 of this 2-part series, Chris Pliakas, the director of Content Services Engineering at Acquia, described how he managed the Acquia Content Hub project, which was released in November, 2015. In this, the second part of the interview, he discusses the Scrum process, the benefits of working with an open source framework like Drupal, and what's next for the Content Hub project.

Q. Are you a new convert to Scrum?

Chris: The first time I was exposed to Scrum was maybe seven years ago. It's just what we did, so I followed along. It's clear to me now that I was going through the motions without fully understanding what makes Scrum work. About two years ago I started digging deeper to learn more about Scrum and the methodologies that influenced it. That's when I'd say I really started using Scrum.

Q. What was the difference, once you started really working with the process?

Chris: The biggest difference was that I could feel the team's happiness. Happiness is often regarded as a touchy-feely subject. Managers might say, It's great if our employees are happy, but we have to ship product. One of my core beliefs is that a team's production level is correlated with their happiness. When you think about it, it makes perfect sense. If you enjoy doing something, time seems to evaporate and you do better work. If you're doing mundane, unsatisfying things, it's hard to slog through it.

Dan Pink, in his book Drive, says that people are motivated by autonomy, mastery, and purpose. Scrum as a framework promotes autonomy by giving teams control over how they do their work. It provides the opportunity for mastery because teams are expected to do everything from soup to nuts, so team members are often stretched beyond their experience level. This sends learning into overdrive, which in turn leads to mastery.

Q. And purpose?

Chris: In Scrum, work is broken down into units called user stories. A key component of a user story is articulating why you are doing the work in the first place. When you couple a short development cycle with the expectation of shipping the work at the end of it, you have to understand each story's purpose so that you can identify the really important things you are not going to do. That's that hard part. We spend a lot of time discussing the purpose of our stories so that we can refine the acceptance criteria to something that can fit within a sprint while at the same time achieve the most important goal.

Q. What is the team's weekly and daily schedule? Are you working on 2-week iterations?

Chris: Scrum experts recommend sprints that last between one and four weeks. We started off with a three week sprint cycle and and then pared it down to two. The quicker you can release the better, so that you can get feedback and make adjustments more quickly. Two weeks made sense for us because the feedback loop was quick enough, but it also gave us enough time to solve the more challenging problems and produce a meaningful product increment.

Q. Do you have a stand-up every day?

Chris: Yes. Same time, same place, every day, no exceptions. One thing we don't do is a formal status update. A lot of Scrum teams that I've been a part of run through the same script: What did you do yesterday? What are you going to do today? Are there any blockers?

Those meetings become really dull, really fast. What you want is aggressive stand-ups. Borrowing Jeff Sutherland's analogy, a good stand-up is like a football huddle where the team comes together to talk about how to move the ball forward. One player might say, I can't get by this lineman, and his teammate might respond with, I can shift over and block him, to open up that hole for you. The wide receiver saying, I ran my decoy route last play -- that would be a waste of time.

That's the nature of our stand-up meetings. When we adopted this approach it felt a little unnatural, but after a couple of weeks the meetings became conversational and laser-focused on moving stories to done. When a team is really hungry and helping each other out, the meetings are actually fun and can break down barriers very quickly. In Scrum all work is visible, so if you really need a status update then go to the scrum board.

Q. How long is a typical meeting?

Chris: We schedule 30 minutes. Some stand-ups last five minutes. The best stand-ups are the ones where someone raises a question that results in a raging debate or a deep help session that takes the majority of that time. We try to cap it at 30 minutes, but sometimes we let things go longer if the conversation is fun and productive.

Q. And those meetings happen five days a week?

Chris: Yes, every day we have a stand-up. It's really important to keep that consistency.

Q. Did it make a difference to the project that you were working in an open source framework like Drupal?

Chris: The benefit of open source is that you get a lot of foundational elements for free. We leverage open source heavily in the backend service that powers Content Hub, and we are transparent about the pieces that we use. Our Drupal integration modules leverage a lot of what's in the community to provide much of the functionality that our customers want. Our job is to tie it all together with a solid experience that is purpose-built to meet our customers' needs. That's the excellent part.

The challenge of working with Drupal is because it's so flexible there are a lot of variables you have to consider. Again, we have two teams. With the team responsible for building the back-end service, we're in full control of our destiny. We own the entire stack and know what pieces we are bringing into the project and how they work together.

With the team that builds the Drupal integration modules, we depend on a dozen other community-maintained modules that innovate and iterate at their own pace. Since the Content Hub modules are installed on sites owned by our customers, we have to account for any combination of configuration across different versions of the modules. The places where things can break increase exponentially with every configuration option. It's a moving target, so we focus a lot of time and energy on automated tests to address the most common combinations.

Q. What would be your advice to people who are taking on a big technology project like this?

Chris: First, I'd say break everything down into small, digestible pieces. This not only promotes shorter development cycles, but it also shows just how much work it is to boil the ocean, which in turn forces you to identify what you aren't going to do.

Second, focus on your deployment pipeline. You need to get the product off your computer and out into the world. In order to do that, you need to invest in the systems that enable you to do it quickly, repeatedly, and reliably.

Third, it's not going to feel right when you release the first iteration of your product because the increment will be so small that you won't think you're adding any value. You will probably get some negative feedback from your initial stakeholders as well, but if you put on your raincoat and embrace the storm you can get a lot of valuable information.

Q. You have to be prepared for early users to rain on your project?

Chris: Building a product usually feels like stepping out into a storm. You are all warm and cozy inside, then you walk out the door and get soaking wet in the freezing rain. It's really tough. Everyone has an opinion, and a lot of people will be quick to tell you why your product is terrible and how much stuff is missing from it. We techies aren't necessarily known for our tact in delivering this type of feedback, so it is impossible not to get defensive when you are on the receiving end of it. However, if you can remove yourself from the emotion there is really valuable data buried in what is said. You just have to be objective and dig a little bit to find it. The only way to separate yourself from the emotion is to constantly expose yourself to the storm, because like most things you get used to it over time. This is what I mean by putting your raincoat on.

Q. The old waterfall model: you wait until the project is finished, and take the risk that you may be walking into a deluge.

Chris: Yes, exactly. I tell my team that we're going to get a lot of feedback, and it will be shared with the team regardless of whether it is positive or negative. At the end of the day, all feedback is good and should be celebrated. We can't risk guessing what our customers want, because no matter how much we talk things over, we will usually make bad assumptions. This doesn't mean we don't know the product or our customers, that's just the way it is. Waterfall only works when your assumptions are right. Scrum, Inc. conducted research that concluded 86% of waterfall projects fail. If I'm honest about how often my assumptions are wrong, this matches up perfectly.

Q. What's next for the Content Hub project? Where does it go from here?

Chris: Borrowing terminology from Crossing The Chasm, we are currently [Jan. 2016] tackling the early market. In my estimation we've made some great progress with the technology enthusiasts and have solid momentum with the visionaries. We've found our niche, and our sales team has done a great job finding customers who are aligned with our vision and core capabilities.

This is when things get exciting. As we look to not only grow our niche but also expand beyond it, it will be interesting to see which direction the market takes us. Although we targeted a specific use case, the foundation is like a Swiss Army knife that can achieve many different use cases, some of which I am sure we haven't thought of. As long as we continue to focus on our ability to respond to change, I see lot of opportunity to solve some really challenging problems that are preventing our customers from focusing on the core of their business.