| Overview: This article is the second in a series of five | | | | organization had a defined group called the |
| which explain how an IT organization delivered a | | | | Architecture Review Board (ARB) which convened to |
| release management process that exceeded its | | | | assess the technical and organizational impact and |
| management's expectations and provided a | | | | risks of major changes to the environment. This |
| foundation for continued success. The series includes: | | | | group consisted of the 5 IT Managers, the |
| | | | Applications Architect and the Operations Architect, |
| 1. How did we get here - THE CONTEXT | | | | and occasionally the CIO. As part of our definitional |
| 2. First solution steps - DEFINITIONS AND TRIAGE | | | | work it was determined that this group would |
| 3. Intake and Release Planning - THE CORE | | | | exercise an expanded role - to quickly and routinely |
| SOLUTION | | | | review each incoming change request. The Release |
| 4. Production Change Control - FINAL QUALITY | | | | Manager was added to the Board. This board was |
| CONTROL | | | | the "neck of the funnel" for all new items and |
| 5. Metrics and Insights - LESSONS LEARNED | | | | through discussion, they determined rough size, |
| Summary: Many Information Technology organizations | | | | priority, and impacts. The ARB also made the specific |
| flounder when they are tasked to understand, | | | | assignment of a SPOC to each new Change Request. |
| organize and implement numerous changes to the | | | | More on the role of the ARB is covered in Article |
| system and application software serving their clients | | | | Three- Intake and Release Planning. |
| and end customers over a period of several years. | | | | Release Planning Group (RPG) The primary |
| This second article focuses on the very first steps of | | | | organizational element that needed to be set in place |
| the solution I developed for the Release Management | | | | was a new group titled Release Planning. This team, |
| consulting engagement. Please refer to the first | | | | facilitated by the Release Manager, met with great |
| Article - THE CONTEXT for a full discussion of the | | | | discipline and regularity to organize, re-organize, and |
| problem domain and organization. What was the | | | | commit to a comprehensive, concrete order of |
| secret sauce that made Release Management a | | | | implementation for all Change Requests. While this |
| success? The answer begins with core problem | | | | sounds straightforward on paper, remember that the |
| analysis, proposed solutions and a triage effort. | | | | context for this role was to organize +/- 325 Change |
| Core Problem Analysis: The company and its IT | | | | Requests into a series of planned releases - and do |
| department clearly wanted a solution implemented | | | | this repeatedly as new things got added each week. |
| that avoided historical mistakes. As a retained | | | | This was a puzzle with ever-moving parts. The |
| consultant, I conducted a series of one-on-one | | | | Release Planning Group consisted of the 5 IT |
| interviews, examined remnants of Change Control | | | | Managers and the Release Manager. The Release |
| documents from a predecessor, investigated | | | | Manager published the current Release Schedule as |
| commercial off-the-shelf packages for both | | | | an outcome of each of the group's meetings. |
| processes and operational tracking, and discovered | | | | Change Control Board (CCB) This group was chaired |
| the following core problems. | | | | by the Configuration Management leader, and had the |
| Problem #1 - What Trees are in this Forest? A | | | | responsibility to review and approve or defer the |
| substantial source of confusion and discussion | | | | completed Change Requests for implementation in |
| involved defining the scope of what things should be | | | | production. The Operations Manager and QA Manager |
| controlled under the umbrella of Release Management | | | | played strong roles within this forum. The SPOCs for |
| and Change Control. There were divergent opinions | | | | each Change Request were questioned for |
| about whether to include/exclude major projects, | | | | preparedness items, including the advance notification |
| infrastructure changes from operations, bug fixes, | | | | of the client communities. The CCB made a |
| hardware changes, Customer service changes, | | | | consensus decision on each Change Request and the |
| emergency patches, etc. As with most debates | | | | outcome of these decisions allowed the Configuration |
| within IT, the stakeholders frequently used similar | | | | Management Team to prepare the scripts and code |
| terms to mean very different things. Very specific | | | | packages for production upgrades. |
| language needed to be written, socialized and | | | | Problem #3 - How To Introduce Order Upon |
| implemented that reduced this ambiguity and | | | | Chaos?-Solutions This fundamental problem afflicts all |
| confusion. | | | | business organizations. Customer requirements |
| Problem #2 - Who is Responsible for What? Projects | | | | constantly change in nature, new ones are added, |
| were very clearly controlled end-to-end by Project | | | | and old ones wither yet refuse to die. Progress on |
| Managers, and at any given time the 4 PMs would | | | | the production line in IT is swift, stuttering, |
| have 2-5 projects underway. These generally | | | | under-resourced, or overwhelmed. Managers |
| covered scope for multiple applications and multiple | | | | independently made decisions from their own |
| person-months of programming effort. Beyond that | | | | perspective of the best interests of the company. |
| good start, Client Change Requests, Bug Fixes, | | | | Frequency of change was sporadic. To introduce |
| Operations infrastructure changes, Customer service | | | | order, the Release Manager defined a disciplined |
| changes (move/update/fix my workstation) and | | | | business cycle for Release Management Processes. |
| emergency patches had no one role identified for | | | | The business cycle was a repetitive set of scheduled |
| control, communications and accountability throughout | | | | meetings of the Architectural Review Board, Release |
| their life span. The IT assembly line for such work | | | | Planning Group, and Change Control Board that would |
| was disjointed at best and lacked fundamental | | | | be executed with defined agendas and deliverables, |
| structure. The role of Release Manager itself was | | | | without failure, and with full participation of the |
| undefined, with various stakeholders holding unique | | | | people in their assigned roles. This business cycle |
| viewpoints on the scope of the assignment. | | | | would commence as soon as a triage of the Change |
| Problem #3 - How To Introduce Order Upon Chaos? | | | | Request queue of work could be completed. |
| This was a very critical concern, as the inflow of new | | | | Customer feedback loops were defined for each |
| requests from the business could not be halted, due | | | | stage of the process. Triage was a critical first step |
| to political and practical matters. If the organization | | | | and is discussed further in this article. The plan for |
| knew tactically where it stood on Monday for every | | | | this business cycle was presented to the CIO. She |
| item, by Wednesday the landscape had changed, and | | | | approved the plan for this business cycle, committed |
| priorities for older work were being adjusted | | | | her management team to its principles, and |
| on-the-fly, either overtly or covertly by business | | | | successfully sold the plan to her peers in the |
| leadership. | | | | company. |
| Problem #4 - How Frequently Should We Release | | | | Problem #4 - How Frequently Should We Release |
| Change Packages? A practical concern was whether | | | | Change Packages?-Solution The debate on this was |
| IT should embark on weekly, biweekly, monthly, | | | | not difficult - once we had made the earlier decisions |
| quarterly or other frequency of planned production | | | | to include operations change requests and exclude |
| software change. The frequency of change would | | | | the emergency software fixes. We settled on a |
| end up driving the timing of the real-world series of | | | | 2-week release change cycle. Our internal customers |
| gates and meetings necessary for control and | | | | were already seeing changes made (or attempted) |
| adjustment. The first solution proposal entailed | | | | weekly with mid-week exceptions and surprises, so |
| purchasing a complete software application and | | | | this could have been viewed as a step back by IT. |
| documentation package from a market leader that | | | | However, the IT managers saw many shortcomings |
| promised to cover the full scope of their interests. | | | | with more rapid attempts at change and were far |
| The alternative proposal was founded upon a low | | | | more confident that the company would be |
| level of automation, and a high degree of | | | | well-served on a 2-week cycle. Specifically, change |
| inter-personal collaboration to achieve similar ends. | | | | requests would be bundled together into a Release |
| Conclusion/Transition The CIO, facing this situation, | | | | Change Package for implementation on alternate |
| agreed to allow the Manager for Project | | | | Thursday evenings. Our fallback position, if the |
| Management and Analysis to contract for a resource | | | | Release did not succeed on Thursday night, would be |
| to implement Release Management (Version 2). The | | | | to switch to a Friday evening implementation. |
| CIO believed that she could deliver better results to | | | | Triage At this stage, the CIO evaluated how quickly |
| her constituency by implementing change in a series | | | | all this good foundation work could be put into |
| of well-understood application package upgrades at | | | | operation. As Release Manager, I advocated for a |
| regular intervals. She also wanted to take back to her | | | | process solution supported by an industrial-strength |
| peers a plan that they could understand and use to | | | | commercial application that could be leveraged |
| directly influence the order of implementation for | | | | toward portfolio management, with many people |
| their changes. The Manager of PM retained me as | | | | updating their component parts, and project-specific |
| the Release Manager with the mandate to institute | | | | support and tracking. However, finding, funding, |
| the processes and controls needed, and engaging all | | | | purchasing and implementing such a baseline tool |
| IT staff and VPs in business departments as needed | | | | would require an estimated 3-4 months under ideal |
| for success. The rest, they say, is "agile" history. To | | | | conditions. The CIO opted to proceed with the |
| learn what it really takes, our story continues next | | | | alternative "low-tech" approach for her organization. |
| with DEFINITIONS AND TRIAGE. | | | | The mandate was to "find a way" to implement the |
| Definitions I will remark on Definitions first, because | | | | essential processes by using the lowest-budget |
| the management team needed to ground itself and | | | | approach. The mandate was daunting. The prior |
| communicate in a consistent fashion about the key | | | | "Change Coordinator" person had worked from a |
| objects and controls with Release Management | | | | Change Control log in Excel that had fallen into disuse. |
| processes. | | | | The root causes for that condition appeared to be |
| Problem #1 - What Trees are in this Forest? - | | | | that the information could never be kept current, plus |
| Solution We took the perspective that anything that | | | | it only covered some of the Change Requests. No |
| is planned to change the configuration of the | | | | one had previously been assigned the responsibility to |
| production computing environment within the | | | | actual know and communicate the status of "small" |
| controlled data center and network configurations | | | | change requests - of which there were several |
| was subject to Release Management processes. As a | | | | hundred. The log also attempted to store a lot of |
| result, we included: | | | | interim dates on small changes, and it duplicated |
| · Application software changes requested by | | | | interim dates that Project Managers maintained in |
| clients or from IT itself (re-factoring, etc) | | | | MSProject on regular projects. The SPOC role had |
| · Application software fixes that were "not | | | | not been defined. As far as we could tell, 285 Change |
| immediately damaging" clients' business | | | | Requests were "open" - meaning submitted by clients |
| · Each software change package from | | | | and not yet completed. That number fluctuated each |
| projects (projects typically had more than one | | | | week as new ones got added and some got finished, |
| release package over several months) | | | | but no one was certain of the status of each item. It |
| · Network or server infrastructure upgrades | | | | seems appropriate to define the term triage |
| (OS, DBMS, middleware, hardware, etc.) | | | | (courtesy of dictionary.com) -noun |
| We excluded from Release Management processes: | | | | 1. the process of sorting victims, as of a battle or |
| · Customer Service requests (fix/move my | | | | disaster, to determine medical priority in order to |
| workstation, office moves, etc) | | | | increase the number of survivors. |
| · Emergency production software application | | | | 2.the determination of priorities for action in an |
| fixes (fix it now) | | | | emergency. |
| This last exclusion was troublesome, but necessary. | | | | The IT managers knew we couldn't cope much |
| We assigned total responsibility for managing the | | | | longer with incorrect information about all the victims |
| emergency fixes to the Software Development | | | | (Change Requests) that were littering our battlefield. |
| Manager, and set an overall target to keep these to | | | | As Release Manager, I asked them to devote the |
| fewer than 5% of the total changes made into | | | | resources necessary to obtain a current, accurate |
| production. (We tracked the numbers, but didn't | | | | view of the following for each Change Request: |
| stand in the way). The practical outcome of these | | | | |
| agreements was that each individual thing included in | | | | 1. Change Request Number |
| Release Management was a Change Request, to be | | | | 2. Customer Name |
| initiated with a simple form. Each would be assigned a | | | | 3. Change Request Label (very short description) |
| unique Number (and key attributes) and controlled in | | | | 4. First Requested Date |
| immediate form by an Excel spreadsheet updated | | | | 5. Status (Open, Completed, Being Worked, |
| and distributed by the Release Manager. | | | | Cancelled, or Duplicate) (if completed, wanted a |
| Problem #2 - Who is Responsible for What? -- | | | | completion date) |
| Solutions | | | | 6. Target Release Date (Not Available was OK) |
| Single Point of Contact (SPOC) The organization had | | | | 7. SPOC Name |
| a good model of behavior and accountability for | | | | I was responsible for facilitating the triage process. |
| projects, but there was disjointed accountability for | | | | This roughly consisted of some very long all-manager |
| all the other Change Request types. To solve this, | | | | meetings, publishing many versions of a new Release |
| we defined a role called the Single Point of Contact | | | | Change Request log in Excel, assigning segments of |
| (SPOC). The role was accountable for conveying the | | | | the list to the most knowledgeable workers for |
| requirements, correct status of IT progress, and | | | | update, and numerous interventions. The sorting |
| sponsoring the Change Request for its ultimate | | | | process consisted of IT managers agreeing on an |
| release to production. The SPOC was individually | | | | initial High, Medium or Low priority for an "early" |
| accountable for telling our clients the timing and | | | | Release Target per Change Request. Customer VPs |
| impact of the implementation of a change, so that | | | | were also polled on their priority settings for Change |
| our clients were adequately prepped for a release. | | | | Requests. The process was declared "finished" in 3 |
| The assignments to this role were expected to last | | | | weeks. We achieved a stable state of Change |
| for the duration of the Change Request, as opposed | | | | Request information in the log and were ready to |
| to the previous pattern of shifting responsibility. As a | | | | begin Release Management processes and the |
| practical matter, the SPOC assignments for 75% of | | | | business cycle for control. The saga of Agile IT |
| the Change Requests fell into the Project Manager | | | | Release Management continues with THE CORE |
| Business Analysis team. | | | | SOLUTION. |
| Architecture Review Board (ARB) The IT | | | | |