At a couple of recent conferences around DevOps (MSP DevOps Days and DevOps Enterprise Summit), Heather Mickman and Ross Clanton from Target explained some high level concepts and actions we’re taking to break our mold of working on complex problems. When presented with an opportunity to work differently and challenge our normal delivery of IT assets - from plan to decommission - what would that look like?

First, we asked some questions:

• What can we do to address the barriers within our organization; can we remove that impact on our ability to get things done?
• What can we do to remove time spent moving from meeting to meeting, often recapping the same discussions that you had in your last meeting 2 weeks ago?
• What process barriers exist that can be included in the delivery and built as part of the ‘service’?

And then we asked … What if we could alleviate all of these problems (and more) all at the same time, all while you build team morale and actually have fun (gasp!) building stuff?

What if?

Two simple words that can: spark amazing conversations; generate innovation and excitement; challenge the status quo; break through to show how DevOps philosophies break through - even with the big horses. A ton of this post is simply about getting stuff done (GSD) but also to get into some specifics around how we approached fixing a business problem in some new ways - at least for us.

So let’s embark on some early details around Target’s journey to challenge the traditional approaches to get stuff done and take control of the ‘how’ … to redefine our working philosophies, principles, and minimize the impact radius of our efforts while still going deep with the teams and persons engaged. Any endeavor such as this has to have a name …

We affectionately call this process FlashBuilds!

What is a FlashBuild?!

Glad you asked!

FlashBuild: A one-day session that includes 2 agile sprints, all the smart people that you need to build stuff, a facilitator, and a whole lot of positive focused energy. You will solve difficult challenges faster than you thought possible, build amazing new products, improve lines of communication, and build relationships between different areas of your organization. Don’t get me wrong, not every FlashBuild results in a resounding success. Sometimes they fail to produce a finished, viable product. They never fail to produce incredible learnings and a greater appreciation for what your partners in the organization are working on. There is no such thing as a failed FlashBuild if you learn something and have fun!

A little deeper FlashBuilds are comprised of three main components:

• Flash mob
• Scrum
• Hackathon

The name originated as part of a project to deliver end-to-end services around API enabled infrastructure. The goal was to leverage our infrastructure automation assets and implement self-service catalog products that our clients could consume - from cradle to grave. Subscribing to the philosophy of ‘minimum viable product’ (MVP), we iterated through defined releases - the next building off of the last - to create a rapid deployment model of new functionality. The service wasn’t just spinning up a VM and send login details to the requestor - that didn’t define the service delivery enough. Early on, we identified there was more to the MVP. We needed to enrich the communication between the service catalog and the back-end systems to avoid ‘request black holes’ to provide our clients with near-time visibility into their requests. We also needed to identify metrics for capacity and service health. And finally, provide actionable life-cycle events into our CMDB, from creation to retirement.

As we expressed these additional challenges, we articulated the requirements to our respective teams and looked at this effort not as a project but in a product model.

The setup - what do we need?

• A business problem
• Team members close to the problem to provide context
• Team members to participate in (sometimes) strange, foreign roles
• Passionate and engaged leadership committed to finding better ways to get stuff done

A Business Problem

The business problem will usually be defined in the context of a technology issue or obstacle. This is an easy thing to grasp when we approach the FlashBuild as ‘one part hackathon’ but process is just as much a sinking anchor to getting stuff done as obstacles around technology. We know that technology moves oh so very fast and process is an entry point for decay and entropy - and to lull people into a mindset that ‘the work today is good enough for tomorrow’. The business problem above, self-service end-to-end API enabled infrastructure deliver, enabled a core group of team members, enthused leadership, and stakeholders to rally and focus on new approaches that incorporate Agile mindsets and incubate this new method of FlashBuilds.

Close in Proximity

One of the aspects to FlashBuilds is proximity, which has at least two meanings - co-location of core team members close to the problem and short-access times to experts for fast feedback. We quickly identified the need to have team members in the same room during the effort but also needed to have close access to our system and process experts to bust through constraints that normally hamper progress … you can hear phrases when these things happen: ‘best practices…’, ‘the way we do it…’, and ‘standard process is …’

Our challenge back to that thought process was really pretty simply: ‘What if …?’

• What if you didn’t have that process?
• What if we could try something new?
• What if that constraint didn’t exist?

What would it look like?

By flushing out the counterfactuals (we should, could, would do /that/ a different way), we typically identified with things in different ways that would allow us to, collectively, see what used to be must haves as nice to haves and our list of deliverables for an MVP would shrink.

Playing Roles

Many times, team members in the room play roles that were very new. This can include being a product owner, tech lead, facilitator (or scrum master), etc. The path we setup to the FlashBuilds follows a scrum approach to managing workflow - without getting too far into the weeds about terminology or applying pure scrum to the day’s activities. This is the ‘flashmob’ part of FlashBuilds. The teams are cross-functional and empowered to make decisions around the goal at hand. This empowerment is enabled through empathy, collaboration, and a safe place to test-and-learn (tech, process, and participating in new roles).

The product owner is the one person identified with accountability of helping to clarify the MVP. An essential aspect of this role is being able to articulate ‘DONE’ to the team with acceptable iterations - with limited timing for each FlashBuild, the end of day goal had to be both usable in helping address the business problem, functional for the product owner, and an obtainable delivery for the FlashBuild team.

The tech lead will fluctuate based on the technical or process focal point of the FlashBuild. This includes a shift of this role between different team members during the course of a FlashBuild. Expertise in the room helps to define this, not titles or pay grades.

The facilitator (or scrum master) keeps the room moving and on schedule. There are also times when this person may be tasked with helping address spikes and blockers the tech team identifies. For example, helping to gather team member resources from teams that aren’t in the room but some detail is needed from that external team to answer a question, verify information, or coordinate scheduling.

Leadership Drives the Culture

A critical component of enabling FlashBuilds is having leadership support - this include support of the end goal and to trust the process to complete said goal. This can be counterintuitive to some management philosophies with incentives focused on piece-work (closing tickets, for example) with past experiences that focus on controls around both ‘what’ to do and ‘how’ to get things done. When introducing foreign concepts, establishing clear feedback loops, an open environment for participation, and following through on commitments is vital to a well-orchestrated FlashBuild. Managers that ‘give up’ a team member for a day have to feel there’s some amount of compensation coming back into the respective area. This compensatory return can be in the form of new consumers to a platform, a team member able to share back with a home team lessons learned, or a new service/function of an IT asset.

The Setup of a FlashBuild

As mentioned above, FlashBuilds are setup as a one-day session … with an end of day goal.

We co-locate into one room. Preferably a room with windows, a big (BIG!) white board, enough room to spread out, and A/V equipment for both in room music and closing FlashBuild demos.

The structure for the day is somewhat typical to other ‘all day sessions’:

• Gather at 9:00 am for initial planning :: This is the time for product owner to establish the business problem and answer basic questions
• Planning at 9:15 :: User stories are identified and an MVP is established
• Working session at 9:30 :: The core team breaks down the user stories and gets to #making_awesome_happen
• Stand-up at 10:30 :: Adjustments are made based on any issues or concerns identified

This is really important as there may be a couple of smaller teams within the core group working multiple fronts (think a tech track and a process track) that have interdependencies

• Working session at 10:45
• 12:00 lunch break & close of Sprint 1
• Stand-up at 1:00 & start of new Sprint :: User stories updated and planning updated
• Stand-up at 1:30 :: New sprint and user story merged with backlog
• Working session at 1:45
• Stand-up at 2:50
• Working session at 3:05
• Demo at 4:00 PM
• Retro immediately following

Ahh… the demo! This is the opportunity to close loops, collaborate broadly, and get people invested irrespective of their perceived role or contribution. The demo becomes the spot where ‘the rubber meets the road’ as part of the FlashBuild. The demo is the MVP in a real, tangible form.

There are many similarities to Agile workflows: scrum, sprints, kanban boards, etc. The condensed implementation of Agile immerses people into the process without having to get too overwhelmed with terminology. In fact, many times the terms aren’t really that important or even socialized until team members start to ask questions.

For example, kanban boards. This is a big deal in FlashBuilds. We use a hybrid approach for kanban. We still have three main sections:

• Backlog
• WIP
• Done

How do things differ? We don’t, out of the gate, normalize a rule set against how work flows through kanban. We keep things free flowing and task oriented at a level our core team is ready to consume. After all, if the team using tools like kanban to visualize workflow aren’t comfortable with the tool, reservation and distraction will come to light with concerns like: ‘Am I using it right?’, ‘What if I do it wrong?’. We start small and build up as new people get more accustomed to the tooling to render workflow, backlog, and ideas of spikes and blockers.

To simplify for folks even more, the WIP portion is more tactile. We use LOTS of paper notes - the kind with a little strip of adhesive - during FlashBuilds. WIP is taken from the Backlog to where ever you are working. You take it with you - as a reminder when distracted or as an announcement to others what you are working on right now. Once complete, the note is moved to Done. We’ve experienced examples where too much work gets put into WIP and that can quickly ‘scrum-fall’ you into getting NOTHING to Done. To counter this early on, you take your WIP note with you and work that to Done.

How About Outcomes?

We’ve used FlashBuilds for multiple business problems - technical and process alike.

For the self-service, end-to-end API friendly provisioning, we completed MVPs around server provisioning, including ITSM governance requirements, in a few full day sessions. This included portal access to instantiate, report, and deprovision workloads all with ITSM governance built into the process.

After a few more sessions, we leveraged the same framework built into the first release, to add self-service Apache and Tomcat options. Soon, we had highly available instances using HA Proxy. The last iteration was a full stand-alone Oracle database on a single VM.

Any of these deliverables complete in less than 20 minutes - in fact, our HA Tomcat would complete in under 8 minutes under normal conditions. A single VM would be complete in about 4 minutes. With ITSM governance built into the full life-cycle, there was no cumbersome change control form process or superfluous steps for a client or consumer to have to execute on to get to their assets. Having the ITSM experts in the room and part of the core team closed feedback loops faster and put resources in the conversation to clarify ‘must haves’ vs. ‘nice to haves’.

To deliver on the service side of this delivery, we leveraged our CMDB for all the IT assets. But went further than just providing metadata about the specific servers. Since the service was comprised of multiple platforms and technologies, we had to provide instrumentation and metrics about overall capacity and health of the environment and present that in a similar manner through the same portal to keep user experiences as close to the same as possible.

What is a FlashBuild Not?

We’ve tried to be very careful with where we employ the FlashBuild process. This is not a SWAT type tool … FlashBuilds could be but there’s arguable more rigor and discipline in Agile (i.e. sprints, demos) and that is most certainly an aspect that many team members find surprising about FlashBuilds. Also, if documentation of a process or technology were an end of day goal, the FlashBuild process probably wouldn’t be the best way to clear the documentation problems. However, based on the outcomes of a SWAT or documentation working session, there may be specific business problems that a FlashBuild approach could help in addressing.

Thanks to Target Team Members

To launch this type of effort, there are multitudes of different people that play significant roles in getting this type of product launched. I can’t remember everyone involved in every FlashBuild and the process of FlashBuild evolves and improves through every execution. A symbol of the strength of a process is the fortitude and endurance of those participating to not use FlashBuilds as an ‘in the moment experience’ but to take with them lessons learned on ways to execute against complex tech and process issues in her or his working day. It’s not by mistake or luck but effort and hard work that FlashBuilds improve.

By understanding broader systems-thinking approaches, different collaboration methods, and experiential learning, those who facilitate FlashBuilds find ways to improve (#KAIZEN) the FlashBuild product. Participants walk away DevOpsing … and sometimes, they didn’t even know. The most common outcome is minds open to different working models and comments like, ‘All of our work should be like this!’ and ‘When’s the next one to work on problem <______>?!'

But if not for the Target team members who support, facilitate, and participate - with open minds and a desire to improve - the product of FlashBuilds wouldn’t be as successful as they are.

Special thanks to Steven Bauer for helping me write and proofing this post.