A lot has been written about product prioritization in both small teams and large product organizations. Much less has been written about what goes on in between. Hence, we want to share our experience in growing from just founders to a group of 40 engineers, data scientists and product people at our last startup.
When Matt and I started x.ai with a couple of other co-founders, choosing what to do was easy. We had a clear, singular mission to build an AI agent that does one thing - schedule meetings. We were building something from scratch. We had high alignment and there was no tech debt, bug list or customer requests (also, no customers.) Our one formal meeting was when the founders got together Monday morning and planned our work for the week. Every week we discussed the list of items we need to build in order to prove that our AI agent was something the world needed. We were constantly in touch with each other throughout the week. We were of one mind without the need for any additional process.
Not just founders anymore
A year into the venture, we got some very enthusiastic users, raised a bit of money and staffed up the tech team to 10 engineers and 5 data scientists. We had no dedicated product people and the founders drove product vision and the day-to-day execution. The team was mostly organized by functions (e.g., frontend, backend, data science) and work was planned within that structure. For the engineers, we settled on a traditional agile process with weekly planning meetings and daily standups. Depending on the function, product priorities were either crystal clear or a bit murky. Our understaffed frontend team was building the basic web UI needed to support essential customer requests or the needs of our growth team. This team worked with me and the growth team to decide what to build every week. Our backend/business logic team’s priorities were more complicated. At the time, x.ai’s design principle was to mimic the interaction of a great personal assistant. To learn the many use cases (and there were MANY) and generate a large and well-labelled data set for machine learning, we created a new role - AiTrainers. The AiTrainers processed incoming emails and labelled the text using our proprietary in-house webapp. Our backend team would often work side-by-side with our AiTrainers in order to extract and automate the most frequent use cases. Prioritization was done through intuition of the AiTrainers and the engineers. Because of the high level of interactions between the groups, the approach was messy but effective. We measured the number of automated use cases and saw the percentile went up rapidly every week.
First attempt at autonomy and transparency
We blinked and our tech team ballooned to over 30 in the fall of 2016. I was quite stubborn about keeping the team flat and preserving individual developers' autonomy and decision making power. The team remained fairly siloed, with different functions owning different parts of our stack. Communications got exponentially harder as the team grew. Many decisions and projects are cross functional with different teams having different priorities and concerns. Basically, we weren’t consistently good at prioritizing our work. One great thing about our approach to management was experimentation and iteration. We began experimenting with this Product Prioritization System pioneered by Pandora. Every quarter, we asked the whole company "why the company would be dumb not to invest in a particular idea or feature in the next three months." This question generated 50-60 ideas each time. I would sit together with a few key members of the tech/product team and estimate the size of each idea. After the estimation, cleaning and consolidation of these ideas, we brought the company back and voted on each idea just like Pandora. We followed this system a couple of times with mixed results. It was better than our existing approach. The team was more brought into our plan and people whose ideas didn't get selected understood the reasons. We eventually abandoned this system because of a few major issues:
People voting for the ideas had mostly their own domain and priorities in mind which let to local optima when choosing what to build (e.g., data scientists and engineers banded together to vote for projects that advanced machine learning techniques, regardless if that was our most pressing company challenge)
The boring but critical improvements needed by our customers didn't get chosen. I ended up having to mandate them anyway
Overall, people felt overwhelmed by the process. Many didn't feel they have the context to make this type of decisions for the company and the selection ended up feeling random
Second attempt: cross-functional, autonomous teams
So the experimentation continued. Taking into account the feedback from past attempts, we settled into a structure that I love - small, autonomous, cross-functional teams. In the beginning of the quarter, the management team would agree on a set of Company OKRs. Then, we divided up the team into 4 to 5 Workstreams which consisted of a product manager, a tech lead and 2-3 developers plus maybe a designer. Each Workstream would work with me to create their team OKRs which tracked back to the Company OKRs. The composition of the Workstreams were evaluated quarterly as well. Sometimes, the Workstream stayed intact. Other times, we created new Workstreams and closed the old ones. The Workstreams planned their work against two 6-weeks cycles per quarter while continuously deploying to production as features became available. Internally, the Workstreams ran 6 1-week sprints within each cycle. The Workstreams had autonomy in deciding what to build and how to build them as long as they were advancing on the team OKRs. We found that a commitment to hold priorities constant for 6 weeks was the right balance between the dynamic environment we were in and the need of our team to ship substantial pieces of software. The team-level OKRs were the right level of abstraction, giving the teams ability to impact their numbers without needing to keep context of everything that was happening in the company.
Of course, we didn't get many things right the first time. The team took months to learn to run an efficient agile process. We didn't have enough product managers to dedicate to every Workstream. We didn't fully solve the balance between advancing high value features and serving foundational modules not under active development. The Workstreams were great at building but not amazing at maintaining. We did establish functional groups that were charged with maintaining key systems as well as pushing best practices. Though in reality, the push for delivery within the Workstreams was so strong that maintainer tasks often took a backseat.
Though not perfect, if your team has reached 15 to 20 developers, I would recommend that you consider organizing your team as small, autonomous, cross-functional teams. When you pair these teams with clear, well aligned goals, they are incredible!
If you are nerdy about productivity like us, come check out NorthShore.ai. We are here to help you organize tribal knowledge, shape the habits to keep them maintained and improve your onboarding/offboarding process.