Incremental Change in E-Commerce Product Development
By Konstantin Ribel – Certified Creating Agile Organizations Trainer.
Preface
This experience report provides an insightful look into the gradual transformation of an e-commerce platform’s product development organisation. Without resorting to major structural changes, the report details a series of incremental and strategic modifications aimed at enhancing team dynamics, collaboration, and overall efficiency. Readers will journey through the challenges and solutions encountered in product development and engineering. This report serves as a practical case study in making organisational changes to foster better flow and productivity, offering valuable lessons for those in similar environments seeking to drive change through incremental steps.
Background
A client developed an e-commerce platform. The company underwent a recent, major reorganisation, with many changes in a short period, which created tension, much overtime, and too many changes for the given time for each employee. It seemed an imbalanced, radical change. Consequently, the people were exhausted and the pain of the change was still “in the air.”
My engagement was initiated and coordinated by the director of Product Development (see Figure 1). Consequently, my primary focus was on Product Development, and my secondary focus was on Product Management and Product Engineering.
Our initial discussions resulted in the underlying message: we want to improve our performance —time from idea to monetization.
Discovery To Development Dynamics
The department structure is shown in Figure 1.
Figure 1: Department structure
Product Discovery provided a market view and discovered the target group’s demands. It contained the role Product Targeting Manager. The department usually worked as a single-function group and regularly had to synchronise its work with Product Management.
Product Management translated the target group’s demands into feature concepts. This department contained the so-called Product Owners, who were assigned to one Product Development team (exceptionally two teams). So-called, because their internal role description deprived them of product ownership and resembled business analyst activities, which, in the middle of this endeavour, created some tension between Product Discovery and Product Management.
Product Engineering delivered the technological foundation to develop the product and ensured technical excellence. Most experienced developers were concentrated in Product Engineering. Product Engineering was organised in cross-functional component teams (teams who focus and have authority to work on one architectural component, for example, the shop admin).
Product Development developed the functionalities in cross-functional teams. It was mostly structured as 8 Scrum Teams with one “Product Owner” (PO from Product Management), one “Product Backlog” per team. Each team was cross-functional and cross-component, with designated single-function roles per team member. For example, two front-end, two back-end developers, one UI designer, one quality assurance engineer in each team.
Worth mentioning, because it is rather unusual, there was a detailed and clear product vision, which provides a clear direction to the entire organisation and helps with everyday decision-making
The Problem
One of the company’s problems was an extensive time from idea to monetization. The observed behaviour can be described as: Everyone is busy and doing their best, working efficiently on their task, yet the system is delivering slow and not delighting the user.
One of the causes was a fragmented and dis-aligned organisational structure. When the organisational structure is aligned, there is flow and ease. The structure is balanced so to say. Contrary, in an organisation where the structure is dis-aligned, the efforts spent to move one step further are disproportionately larger, than the value of the step.
Consequently, the organisational structure, in this case, led to task interdependencies between the departments, and a significant amount of asynchronous communications between them to reach cohesive product agreements (scope, design, architecture) for the next steps. Asynchronous communication is the exchange of information where an immediate response is not expected, there is a delay between request and response, for example, chat, email. In contrast, synchronous communication is where the exchange happens in real-time, like in a face-to-face conversation or a phone call.
Figure 2 shows a Design Structure Matrix (DSM, see also [1] [3]). It visualises the department functions (left column) and the department names (upper row). A capital ‘R’ indicates the department which is assigned the function and who is mainly responsible for fulfilling the function. A lower case ‘r’ indicates the required department to help the main department fulfil its function; the departments are dependent.
Key point: interdependencies between organisational units will likely increase coordination costs, goal conflicts, lead to loss of productivity, and, most importantly, lower the ability to respond to change.
Figure 2: Design Structure Matrix showing task interdependencies between departments. [1], [3]
The only independent department was Product Discovery. The potential case, where Product Discovery needs technical feasibility input from Product Development or Engineering, is neglected at this moment.
Product Management was dependent on two departments. When translating the target group’s demands into feature concepts, they often needed to clarify open questions with Product Discovery (r1 in Figure 2). To understand technical feasibility, existing functionalities and valuable small packages, they had to consult Product Development (r2 in Figure 2).
Product Development had three dependencies. When starting work on the feature concepts (translated demands) they typically had to involve Product Management to get information which was lost in translation (r4 in Figure 2). Occasionally, they had to include Product Discovery as the original source (r3 in Figure 2). When planning the next steps forward, they had to agree on architectural changes and UI/UX design with Product Engineering (r5 in Figure 2).
Product Engineering depended only on Product Development (r6 in Figure 2). For Product Engineering to be able to deliver the product’s technological foundation, they had to ensure some quality standards (code and architectural quality). This meant, all changes made by Product Development had to be submitted via Git pull requests to Product Engineering for review before merging them to mainline. More on this later.
The dependencies between Product Management and Product Development (C1 in Figure 3) was not problematic because the “POs” (from Product Management) and the teams (from Product Development) worked closely together and synchronously each Sprint. They also shared measures of success, like customer outcomes, based on their deliverables.
Figure 3: Design Structure Matrix highlighting task interdependencies between departments.
However, the dependencies between the Product Development and Product Engineering departments (C2 in Figure 3) were more severe. Because of the asynchronous working style between those departments, which led to long waiting times in Product Development and high amount of work in progress (WIP).
In the following paragraphs, we will focus on (1) Product Development team-internal dynamics, (2) dynamics and interdependencies between Product Development and Product Engineering, (3) Moving towards Multi-Team Scrum.
Gradual Changes
There was one product for the entire company, a customer-centric product definition, an ideal customer profile, and multiple persona definitions for different user types. This simplified communication between stakeholders, Product Management, and developers; especially, when refining requirements and discussing the beneficiaries of new features. Consequently, the Guideline 1 —Organise in Product Groups [1]— was followed naturally. Guideline 1 suggests to include all people with required roles and organisational elements to fulfil the product’s purpose in one organisational unit – the product group.
Improving things through structural changes was gradual and rather slow (management decision). Therefore, we decided to start working with one team only.
One Team Focus
Team-internal roles led to Tayloristic dynamics in an “agile” disguise – high people utilisation, high single function output, inventory, and WIP coordinated through one “Product Backlog” per team.
“When an organisation focuses on maximising the efficiency of single-skill experts, it becomes difficult for those experts to help others outside their area of expertise. When there is too much work in their area of expertise, they become a single-point bottleneck; when there is not enough work for these specialists to do, they likely start to work ahead. Both situations slow down the development process.” [1]
Usually the team-level “PO” coordinated the team’s tasks between the team members. The Daily Scrum had the underlying message “I worked” with team members focusing on individual output instead of a reflection on the team’s joint progress towards the Sprint Goal. Team members worked according to their single-function role description. Consequently, team self-management was rather low, with team members focusing on tiny technical tasks within their role description, instead of user problems.
Efficiency and Flow: The Impact of Team Structure on Productivity
When a company is organised based on component teams, then there is usually a queue (backlog, todo list) per team. Every team is very efficient working on their specialisation. However, the flow is low. See “CT” in Figure 4. See also queueing theory [1], [5].
This company was organised mostly based on cross-functional teams with single-function roles per team member, which suddenly introduces queues per person not just per team and multiplies the number of queues multifold. Consequently, the flow is still low, however, additionally the person efficiency drops too. See “Q” in Figure 4.
Figure 4: Flow vs. person efficiency. Based on flow vs. resource efficiency [1].
Why? Because the existence of single-function roles motivates the team members to adhere to those as they are also likely being evaluated on their performance within the boundaries of the role description.
Hence, my recommended approach in such circumstances is to remove the single-function roles in cross-functional teams to ease role-free collaboration between team members and focusing on the most important work instead of adhering to role descriptions. This structural element would lead to M-shaped team members or team members with multiple skills, see Figure 5.
Figure 5: Example of a M-shaped (multi-skilled) team member in the automotive industry.
However, removal of official roles was declined. Therefore, a different approach was needed to move closer to Guideline 12: Multi-skilled Development [1]. Guideline 12 advocates to redesign the human operations system (roles, rewards) to make it individually useful to collaborate with others, and balance one’s deep specialisations and broad knowledge.
Mob-Working Sessions to the Rescue
Introductions to Mob-Programming sessions showed that people with different roles were able to work together. In most cases, teams (plural, because I worked with several teams) took this learning into their daily work, and implemented pair-working and learning katas.
This way, teams experienced team-internal collaboration across multiple roles, and the role definitions became less relevant, but not irrelevant. Long-term this remained a structural element, which limited human potential and the company’s performance.
Key point: while a short-term cross-role collaboration and multi-learning is possible, it remains a short-term benefit. Because in the long term, the sheer existence of official role definitions and role-specific performance evaluations will limit the multiskilled learning and true role-free collaboration within a team.
Essential Interdependencies
As mentioned earlier, all teams (part of Product Development) implemented new features using feature branches. Git pull requests were created to merge the branches to the mainline.
Figure 6: Reciprocal interdependencies between Product Development and Product Engineering.
Product Engineering needed to review most product changes, which came from Product Development as shown in Figure 6. The review process was usually channelled via Git pull requests. Since part of Product Engineering’s function was to “ensure technical excellence” —also for work results made by Product Development— both units were highly interdependent. The high interdependence and asynchronous communication led to high inventories of WIP. For more details on interdependencies in this case, see paragraph: The Problem.
One way to reduce interdependencies would be to follow Guideline 3 – Merge Units with Essential Interdependencies [1], which proposes to merge essential units with significant interdependencies, which are required to fulfil a product’s goal, into one larger unit.
However, merging both units was out of the question at this time; partially due to product operation responsibility of the Product Engineering department. Redefining the functions or transferring responsibilities was not in consideration yet, because expertise knowledge was lacking in Product Development.
The Dynamics of Reciprocal Interdependence in Development Processes
The following part is related to Guideline 4 – Combine Authority with Responsibility [1] and Guideline 5 – Contain Reciprocal Task Interdependencies [1].
Guideline 4 points out that in traditional organisational design, the organisation is divided vertically to split responsibility, and horizontally to divide authority. In this case the Product Engineering and Development department were parallel to each other. Yet, the Product Engineering department had the operational responsibility for the product and the final say on the product’s architecture. Consequently, Product Development had the responsibility to develop features, but Product Engineering had the authority to accept/decline changes – a combination of vertical and horizontal division, when looking at the whole Product department.
Guideline 5 suggests redesigning the organisation to contain the reciprocal interdependencies within one unit and manage those through constant information sharing and mutual adjustment.
Let’s look at a definition of reciprocal interdependence by Nicolay Worren: “Reciprocal interdependency means that there is a two-way, iterative relationship, in that the output of one process or unit is the input to the other process or unit, and vice versa.” [3]
The nature of most Pull Requests is reciprocal. Why? Because it is a back-and-forth between the submitter and the approver. The approver provides the feedback, typically asynchronously. The submitter incorporates the feedback and asks the approver to review it again. This practice leads to long waiting times and high WIP.
To address the dependencies between Product Engineering and Product Development, the You-Build-It-You-Run-It (YBIYRI) initiative was created. Its goal was to reduce dependencies on Product Engineering by equipping Product Development with the necessary knowledge and skills, thus shortening the time to “in-production.”
Key point: The overarching scheme was to resolve the dependencies through knowledge acquisition, rather than restructuring. As a result, the original problem of reciprocal interdependencies, and hence the responsibility allocation, as described in paragraph The Problem, remains unchanged. Consequently, this approach can only be a short-term solution.
Our Multi-Step Approach to Higher Focus and Flow
The first step was to shorten the release-to-production-cycle-time. Initially, it was once per month and was changed to once per week. Meaning that everything that was merged in the mainline was released to production weekly.
The second step was to have Product Engineering teams coach and support the Product Development teams. By working together, cross-team learning and trust increased. The first coached team was able to release to production after a short coaching time.
The third step was to temporarily include some individuals from Product Engineering in the teams of Product Development as Travelers [2]. The Travelers joined the teams for one or multiple Sprints. They participated in refinement sessions as well as in multi-team design workshops [2]. They, together, discussed the necessary changes in the product architecture when implementing new features. This way the collaboration intensified, and the communication synchronised. The confidence of changes increased, the risk for and amount of rework after creating a Git pull request decreased. Therefore, the time from feature branch to mainline shortened.
Key point: Involving people whose approval is necessary to finish your work early, increases learning, trust, shortens waiting time, and therefore, reduces WIP.
The perfection goal was to move to full-fledged trunk-based development without any Git pull request.
Multi-Team Scrum
Despite the clear product definition and one product for the entire company, there were multiple team-level “Product Backlogs.”
In a workshop with the head of Product Management (see Figure 1) and all “POs”, we conducted an experiment. We had all “POs” write down everything they considered important for the coming year. The result was many topics, optimising the current perspective of each “PO” — a local optimization, instead of the overarching goals of product management. This triggered the head of Product Management to create a common roadmap and one Product Backlog.
After the initial topics to develop next were decided, two teams were brought together in the B2B Value Area to work within that area. Value Areas, a pattern from Scrum Patterns to respond to unanticipated spikes in expertise demand.
We started with having Multi-Team Product Backlog Refinements and Multi-Team Sprint Planning [2]. Both teams had Travellers from Product Engineering. A Traveller is a person from another team, whose focus is to work with the team, share the team’s responsibility and teach them their expertise to reduce dependencies on themselves, see also [2].
By creating shared communication channels, the teams were able to coordinate and help each other reach the Sprint Goal. If one team was finished early, they helped the other team finish their work before starting anything new. See also: Guideline 8 – Create Conditions for Emergent Coordination [1]. It suggests creating conditions for people to naturally coordinate instead of prescribing rigid coordination processes and tools.
Key point: Multi-team work, emergent coordination and focus on the most important product increments, allows the teams to not only work cross-functional, but also cross-team to reach a common goal.
Conclusion
In this experience report, we explored how we improved the product development process of an e-commerce platform without making structural changes. Instead, we redesigned the process, and people components of the organisation design. Our approach relied heavily on goodwill and incremental changes.
The backdrop was set with an initial assessment of the company’s challenges, notably the prolonged duration from idea to monetization and a dis-balanced organisational structure. This was illustrated with the Design Structure Matrix, pinpointing the crucial task interdependencies and asynchronous communication issues.
We addressed these with gradual changes, beginning with improving the single-team dynamics within Product Development. Fostering multi-skilled capabilities was pivotal in breaking down silos and enhancing inter-team collaboration. The Mob-Working sessions emerged as a significant milestone, blurring traditional role boundaries and encouraging collective problem-solving.
Furthermore, the dependencies between Product Development and Product Engineering was a focal point, where the implementation of the You-Build-It-You-Run-It initiative marked a shift towards reducing reciprocal task interdependencies.
The result of these efforts was seen in the implementation of Multi-Team Scrum, which unified disparate teams under a singular vision. This not only optimised individual team efforts but also fostered a holistic approach towards product development, resulting in improved flow efficiency.
In essence, this report encapsulates a journey of thoughtful and strategic interventions, underpinned by a steadfast commitment to continuous improvement and adaptability. The lessons gleaned here offer valuable insights into the nuanced process of organisational change.
To solidify these achievements and address reciprocal interdependencies, structural changes are necessary. This experience has shown that while goodwill and incremental steps are valuable, lasting transformation requires deeper organisational changes.
References
[1] Cesario Ramos and Ilia Pavlichenko, Creating Agile Organizations – A Systemic Approach, Addison-Wesley, 2023.
[2] C. Larman and B. Vodde, Large-Scale Scrum, Addison-Wesley, 2016.
[3] Nicolay Worren, Organization Design – Simplifying Complex Systems, Routledge, 2018
[4] Jeff Sutherland, James O. Coplien, Lachlan Heasman, Mark den Hollander, Cesário Ramos, and the Scrum Primer Group, A Scrum Book – The Spirit of the Game, Pragmatic Press, 2019
[5] C. Larman and B. Vodde, Scaling Lean & Agile Development, Addison-Wesley, 2009