Assembla Blog: Category Open Source

Lessons for Hyper-Agile product development

andy@assembla.com — Sun, 07 Jan 2007 21:15:00 +0000

Recently we were called upon to build and launch a new Web product with a hard deadline in a seasonal business. By the time we started our work, only three months remained until the planned launch date. It was a big system. Our mission was to build a new and greatly enhanced version of a product which had taken 15 months to build in its last iteration. So, we had a benchmark.

In this case, we ran more than four times faster than a similar project implemented last year with older tools and development methods. We also came in on-time and under budget, and we ended up implementing features faster than we could figure out how to use them.

How did this happen? Want to try this at home? I will share our lessons below. Good luck.

Donâ€™t do things that take time to arrange
Donâ€™t get dragged down by old code
Establish the data schema
Pile on with new team members
Use a ticket list / work queue
Build something now, even though we know we will have to rebuild
Donâ€™t split the codebase into components
Daily releases instead of stabilization periods
Make a usage dashboard
Web services are your friend
Hire users as product managers
Donâ€™t create obstacles: Users provide the data
Documentation on a wiki
Find the one thing people will use

Here is the expanded version, for people who actually do this for a living.

Donâ€™t do things that take time to arrange

We get big time savings by not moving people around. Travel takes time to arrange, and it takes time to do. People should be working, not moving, and they should start now. I donâ€™t do interviews. I found that even arranging a phone interview takes five to seven days â€“ a big chunk of a 70 day project. Instead, I let people start work and I evaluate their work later. I also donâ€™t set up conference calls for the product team. In my observation, time spent on the phone is a good indicator of communication problems, and is best avoided by improved management and communication.

Donâ€™t get dragged down by old code.

I believe in doing a lot of research and trying to adapt existing open source code, as an alternative to diving into new development. We spent a week of our precious schedule taking a in-depth look at open source alternatives. We considered a system with a friendly academic license and $4M worth of development behind it. What we found was big, complicated enterprise software with lots of heavy baggage. We talked to people who were adding to Sakai, and found that they were typically taking 8 months to do things that I thought should take a month. We discarded the existing product for the same reason. We moved on to a more agile architecture and never looked back. I had to change my thinking because, in the new world, it often is faster to build from scratch.

Rails is simpler than java for this type of application, and the build was much faster as a result.

Establish the data schema

It is easy to add to a data schema, but difficult to change it. I sometimes visit old clients and find that the people, products, and code that I knew are long forgotten, but the data schema remains. Data schema is the one piece of up-front design that we do before diving into the build.

Pile on with new team members

It is possible to bring in a lot of new team members and let people pick their own tasks. Using this technique, we were able to triple our output in three weeks. This approach is the exact opposite of the “Mythical Man Month” recommendation, which holds that adding new team members imposes a cost of learning time and management time which will delay a project that is on a short time schedule. In my experience, adding team members DOES delay a project, if you need to guide them and manage them. The pile-on method works if the new developers are good and can find their own way. We are helped by our â€œinspired by open sourceâ€? methods, which were developed to serve large teams with loose central management, and the fact that some of the new developers are very good. We can weed out the rest a few weeks later.

An obvious related lesson is to work with good people. I am not even going to put it in the list of lessons from this project because it is far from a new lesson. A good developer seems to be about 5 times as productive as an average developer. In this project, there were several cases where a team member spent 2 weeks working on something, did not finish it, and then found it replaced by something our best guy finished in 2 days. Given the potential gains, most project managers donâ€™t invest enough in recruiting and retention.

Use a ticket list / work queue

We put all of our tasks, bugs, and new requests into a list of “tickets”, often called a work queue. We use Trac. In fact, we use Assembla.com. We sort the tickets by priority and milestone, so the tasks from the next milestone are on top of the list (queue). Itâ€™s easy to figure out what to do. Just look at the list of tickets and pick something near the top. You can try to adopt a more rigorous agile methodology, but you will find that it is hard to improve on this simple and effective technique.

Build something now, even though we know we will have to rebuild

We started building copies of the old product, and copies of features that we saw in other products, before we even had a coherent visual design or plan for the new product. Then we ripped out a lot of the guts and rebuilt it. This was still faster than waiting around for a coherent design and product plan. Why? In this new environment, development is faster than product management. From a product management point of view, it is faster to ask for changes than to figure out what the product should do from scratch. It takes forever to build something that is unknown, but if you know what you want, builds are fast. So, the two steps (copy and redesign/refactor), with clear instructions for each, are faster than one build from a blank sheet.

Donâ€™t split the codebase into components

In this application we needed several types of servers that were different â€“a permissioned application, a public portal, and a central user registration server. We lumped them together in one repository and one application with configurable â€œrolesâ€? that turn features on or off. This made the code a bit more complicated. However, it made other aspects of managing the app â€“ setting up developers, testing, staging, deployment â€“ a lot simpler.

Daily releases instead of stabilization periods

In a normal Web 2.0 style release, you just run beta releases of increasing quality, until you are ready to do a bigger announcement. In this case, I didnâ€™t think we had time for such gradual stabilization. I initially recommended that we do a code freeze with a week of testing and stabilization. The resulting product might not be perfect, but at least it wouldnâ€™t crash. We started with a daily release process and no code freeze, and it worked well enough that we didnâ€™t need to a code freeze and stabilization period. Itâ€™s not bug free, but we fix the bugs as fast as people notice them. I now have abundant evidence, from several projects, one of them a big enterprise ASP, that users will tolerate a product which comes directly from developers, every day, with minimal testing, if the time to release a bug fix is less than 24 hours.

Make a usage dashboard

Program a custom dashboard that shows registration and usage statistics. Everyone on the team will use it every day as soon as the beta period starts.

Web services are your friend

Arranging integration with external systems can absorb a lot of calendar time. We were able to use SOAP services (easily offered or called in Rails) to make it go faster than I expected.

Hire users as product managers

Product management â€“ figuring out what the product should do â€“ became the bottleneck early on and remains the bottleneck. Using the above techniques, the developers pull requests off the ticket stack and implement them as fast as we can make the requests. Our strategy to deal with this is to hire users as part-time product managers, and let them each work with a mockup team to redesign the visual parts of the app. With the distributed team system, itâ€™s easy to integrate part-timers like this.

Donâ€™t create obstacles: Users provide the data

We took out the centrally controlled categories and allowing users to just type their own tags for the content that they entered and shared. I think we would still be waiting if we had to build our own categories. We got a surprise when we released with a comprehensive list of US data, and we had immediate requests from dozens of countries. I now think that users should be able to post additions to any part of the database if they need to. Why block them? Sometimes the additions can be released directly, and sometimes they may need to be approved by an editor.

Documentation on a wiki

If your application changes every day, you need to put the documentation on a wiki. If you set up the documentation wiki early in the development process, you can start linking to it from inside the app, just like embedded help pages.

Find the one thing people will use

If you can find the one feature that people will use and adopt, it makes the whole system simpler. So, release early and often and watch your dashboard. Find that thing.

Roadmapping - how your product finds its way

andy@assembla.com — Thu, 28 Sep 2006 21:43:00 +0000

A startup will often live or die based on its first product release. Did it get released? Did people find it useful? Good roadmapping dramatically improves your chance of getting to â€œYes!â€? on these critical questions.

Read about my roadmapping technique over at th On Startups blog.

I think it’s my most immediately useful post so far. It’s a technique that has helped me work with individual entrepreneurs, my own products, venture funded companies, and giant corporations. I gave it to Dharmesh at On Startups because he has a much bigger readership that could benefit from this technique.

So, check it out.

Jon Udell Podcast: - A conversation with Andy Singleton about building global teams

andy@assembla.com — Sat, 20 May 2006 22:53:00 +0000

Jon Udell included me in his Friday podcast series this week with a conversation with Andy Singleton about building global teams . Jon has been putting up with me since 1993, when he edited some freelance articles that I wrote for Byte magazine. In 2003 he coined the term “dynamic development” to describe the work I was doing. He has recently been commenting on user innovation and the power of human networks to do work beyond open source sofware. In this conversation, he turns me on to Yochai Benkler’s The Wealth of Networks, and he cautions against a “walled garden”. Maybe we’ll be able to work together on a universally portable user profile.

Project.net - new open strategy and releases

andy — Fri, 17 Feb 2006 12:17:00 +0000

We have finally announced the new strategy and releases for Project.net, at OSBC on February 14. Over the next few months we will release open source, hosted on-demand, and enterprise packages based on this product. We picked up some nice articles in the press.

Internet News
News.com

This relaunch of Project.net has been a long time in the making, since I first discussed it with Roger Bly, the president of Project.net, last May.

At the time, Roger was wondering how Project.net could go to the next level. The company had a nice product and a few good enterprise customers. However, it was too small to thrive in the increasingly brutal enterprise selling cycle, as described in my article Is the Enterprise Software Licensing Business Dying?

I discussed with Roger some opportunities that Project.net could take advantage of:

Many of their best customers were doing some kind of customization. However, Project.net did not have any kind of shared source or open source licensing or process in place to serve those customers. They didn’t have any strategy in place to take advantage of open source marketing, community resources, and customer innovation.
They were making most of their money on the bigger deals, over $100K, and really didn’t have any way to benefit from serving smaller groups.
They built a nice Web-based, multi-tenant application, but they did not offer any kind of ASP or on-demand service.

Over the next six months we worked together to design a new strategy and get the company sold to ICS, which was willing to put in resources to back the new strategy. That deal closed in December.

Then we set to work rebuilding the development team and process as an open-source compatible global team and process. It was an amazing experience, during which we looked at 130 candidates, selected 11 for competitive trials, and were amazed when 9 of the candidates turned out to be great. Now, we have guys from 7 countries working together every day.

The results from the sales and marketing process have also been great. Leads, renewals, and sales are running ahead of expectations. There has been a surge of interest in Project Portfolio Management solutions, and all of the players are benefiting. Project.net has a bright future in this space.

Notes from the Government Open Source Conference

andy — Tue, 18 Oct 2005 15:40:00 +0000

The GOSCON event was a great success, with about 200 attendees, about twice the expected turnout. Below are some observations that might be relevant for for Massachusetts. Linda Hamel, Massachusetts ITD general counsel, was a star of the show. There is a consensus that Massachusetts has set the …

The GOSCON event was a great success, with about 200 attendees, about twice the expected turnout. Below are some observations that might be relevant for for Massachusetts.

Linda Hamel, Massachusetts ITD general counsel, was a star of the show. There is a consensus that Massachusetts has set the standard for open standards policy making. A lot of people wanted to learn more about Massachusetts’ policy, and I spoke with a Red Hat VP who had just gotten back from Malaysia with a report that “they didn’t want to hear about Red Hat. They wanted to hear about Massachusetts.”

BRINGING STATE AGENCIES TOGETHER
About half of the attendees were from various Oregon government agencies, and the crowd included a number of agency CIOs. The event delivered value for the state of Oregon as education, a way to agree on the key issues, and venue for organizing relatively sophisticated joint efforts. I suspect that Massachusetts could get a lot of value from a similar, locally targeted event.

ECONOMIC DEVELOPMENT
Oregon has been developing an economic “cluster” around open source. They view it as a source of jobs and have put some state resources behind it, including official economic development staff with slick brochures. I have two different impressions of this effort. The first is that the number of jobs is very modest – at most 200. If dropped into the larger Massachusetts software industry, the job count wouldn’t be statistically perceptible. On the other hand, the synergies really work and the gears meshed. I met with the CEO of Compiere, who had just moved his company from Connecticut to an incubator in Portland, a short walk from the conference. The conference was organized by the Open Source Lab at Oregon State University, hosted by Portland State, attended by stage and local agencies, and populated with local small open source companies.

PROCUREMENT
There was discussion of procurement processes. The Oregon Department of Transportation CIO stood up and asked “how can I get these guys (Compiere) to respond to an RFP?” One obvious observation is that open source software is so cheap that the vendors can’t afford to respond to RFP’s. By design, the software is bought, not sold. The open source model often requires that users do more of their own evaluations, which requires specific internal skills, or consultants who do research. The requirement to consider open source as a possible route to “best value” is useful to balance this extra effort. Bigger projects will require integrators who should be expected to respond to RFP’s. A two step process seemed attractive, separating software selection from integrator selection. In the first step, the agency is free to select open source software without any bidding process, since that step doesn’t require any expenditure. Then the agency finds three bidders for the integration job. I made a note to work on lining up a quorum of qualified integrators for the more complex open source packages.

SHARED PROJECTS
An explicit goal of the conference was organizing shared projects. Andy Stein promoted the GOCC in a description of his effort to build a portal for local government. I suggested that he open up this project in the development stages and he agreed to consider my position. I met a couple of small companies that support software shared by several states or agencies and want to move it into open source projects. I offered to help with set up the open development process and prequalify some contractors to work on it. This should bring some new inventory to GOCC. We saw shared projects for scholarship administration, medical information, mental health care management, and 211.

James McKerr and Scott Kveton of OSU described their experiences with two significant “community source” projects, Sakai and Kuali, that the OSU open source lab participates in. Sakai is a learning management portal for universities, and Kuali is an ERP system for universities. Both are multi-million dollar efforts. They described the community source model, which involves establishing a heavyweight non-profit organization to manage development, with founding members committing substantial sums of money ($50K-$700K range, $2.8M in initial grants to Sakai) to build a replacement for complex commercial systems that would otherwise cost $1M+. I think this model might be too ponderous for many projects, but it seems to work for the universities, especially as a way to attract grants, it has some successful examples, and it allows for ambitious effort.

Government CIO’s are talking about some very ambitious efforts. For example, Bill Crowell, CIO of the Oregon Department of Human Services, wants to create a community version of a SACWIS system. SACWIS (Statewide Automated Child Welfare Information System) is the system that states use to manage their child welfare cases. The system implements a set of 96 federally mandated processes, plus processes specific to individual states. A SACWIS deployment costs a minimum of $10M, ranging up to an estimated $800M for an upcoming California deployment. There are 11 states planning new or upgraded SACWIS in the next few years.

It turns out that because each SACWIS system is built with federal sponsorship, the code is public domain. However, it isn’t generally shared. The big integrators have their own codebase from their last job that they can extend for a particular state deployment. Each state gets a system that is built to a custom set of requirements by a big integrator and delivered at a particular point in time. Then, because it’s a custom system, there is no upgrade process and no sharing of enhancements, and the system eventually goes out of date. If there were a shared, living version of SACWIS, with a community participating in a continual upgrade process with contribution of enhancements, it would produce a lot of benefits. Initial adoption would be faster and cheaper. The system would be upgraded over time. There would be far fewer replacement cycles. Hundreds of millions of dollars would be saved.

I sat in a working group that included Crowell, Walter McDonald (who wrote the federal requirements and serves as a requirements consultant for most outstanding SACWIS bids), and a gentleman from GGI-AMS, one of the integrators with a relatively modern version of SACWIS. The vendors are hesitant to open up, but based on this type of participation, it seems clear that something can happen.

A lot of federally funded public domain code would be more useful if wrapped in an open source community process, and that’s definitely an area where I want to do more work. Larry Augustin introduced his company, Medisphere, which took the public domain Vista code for managing hospitals (2 million lines of code, $2B invested over 25 years, used in all the VA hospitals) and modernized it for the small and mid-sized hospital market.

I got a decent reception for the idea of organizing a user collaborative for ODF adoption, with responsibility for building an MS-Office plugin for reading, writing, and previewing ODF, as well as for providing other adoption resources and processes. It seems like this would provide a simple route to open document formats for many users. For example, Jeff Kaplan, who just organized the Roadmap for Open ICT Ecosystems study, thought some of his participants (open standards supporters from 13 countries) might be interested.

Sakai - It’s the community, not the code

andy — Tue, 27 Sep 2005 20:49:00 +0000

Sakai is a Learning Management System that provides features for students to interact with their professors and their universities on the Web. Why is Sakai important?

It’s the best example that we have seen of a community coming together to build an industry-specific application.
The community has generated real …

Sakai is a Learning Management System that provides features for students to interact with their professors and their universities on the Web.

Why is Sakai important?

It’s the best example that we have seen of a community coming together to build an industry-specific application.

The community has generated real enthusiasm at both the institutional and individual level. More than 80 dues-paying universities have joined the Sakai foundation, and 400 individuals showed up at the last Sakai conference. More than $5M has been committed to Sakai development so far.

* It’s a category killer. It’s in line to replace a multitude of homegrown systems, and it competes with six commercial products designed for higher education.

I had the chance to speak today with Amitava “Babi” Mitra, a Sakai board member representing MIT. He filled me in on the history and the secret sauce.

The project started when representatives of four universities – MIT, U Michigan, Indiana U, and Stanford – realized that they were each investing in a homegrown learning management system. They pooled their efforts, and by the beginning of 2004 had been awarded a $2.4M grant from the Mellon Foundation. They also began offering an “Educational Partner Program” to other universities, supported by the Hewlett Foundation, and started the process of creating a Sakai Foundation with a more inclusive governance structure.

They began developing an application on open source foundations. Development and beta testing proceeded for a year and a half, and now in the fall of 2005, a handful of universities are putting the latest version into production.

I want to understand the enthusiasm that universities, more than 80 at this point, have showed for this project. They commit $30,000 when they join the partner program – $10,000 per year for three years. What do they get? Why do they join?

They certainly don’t join for the code. The code is available to anyone under a free open source license. Furthermore, the code is still under development and used only in pilot installations. Given that the project has been so successful in the 1.5 years prior to a production release, I almost hypothesize that it is important NOT to have a finished application when you start putting a collaborative like Sakai together.

Nor do they get significant governance privileges when they join. The board is controlled by the four original founding universities, who don’t pay dues and get the majority of the development money.

They get direct support from Sakai’s small permanent staff – training, implementation workshops, questions answered, working groups arranged. Mostly what they get is the membership in the community and a voice in the development process. Community features include conference admission (a $300 value per person), participation in the members-only discussion lists, and the chance to join the working groups (a chance to work for free!). Partners are taking the lead in setting up working goups and discussion groups.

So, universities and individuals are pouring time and money into a product that they can get for free, in exchange for some advice, participation in discussions, and a chance to do extra work. Those are real benefits because they belong to a community. It shows that collaboration is a powerful motivating force.

Mitra gave me his opinion of why universities and individuals are enthusiastic about this project.

It’s a hot category. Universities are just now discovering that there is a demand for learning management systems from their students and professors, and they need to make plans for deployment.
Universities have a bias to control their own code and to share that makes them less enthusiastic about the proprietary products available in this category.
He thinks higher education institutions are culturally unique because they make time for collaboration, conferences, and exploratory projects, and the time and talent wouldn’t be as available in a for-profit institution.

I’m not sure that I agree with the university-specific analysis. People in many industries seem to want to collaborate with their peers.

There are some dynamics that I would expect to see repeated with other industry collaboratives:
The system directly replaces homegrown software
It’s a recent but not totally new category of application that the majority of participants have realized they will need to adopt.
It’s built on open source components
Four founding members seems about right. It’s enough to get real savings, but not too many to form a management group.
It starts with a community and not with finished code.

There has to be an initial event. In this case, a $2.4M grant made the project real and gave it the resources to start from scratch, although Mitra says “it just would have taken longer” without the grant.

On alternative sourcing mechanism that interests me is donation. The Sakai Course Evaluation Working Group is taking an application that Columbia uses to let students evaluate professors (and that Virginia Tech has modified and deployed), and formed a working group around it with the goal of offering it as a Sakai tool.

Democratizing Innovation shows how to make big breakthroughs

andy@assembla.com — Tue, 05 Jul 2005 17:43:00 +0000

User innovation, in the form of open source software, is one of the most powerful forces in the software business today. Will this force propel the software industry to yet another phase of bold innovation? Or, will it cripple commercial R&D organizations and lead to a future of stagnant, commoditized software?

In his book Democratizing Innovation (available online), Sloan School professor Eric Von Hippel shows user innovation at work in many industries, including sporting goods, surgical equipment, electronics manufacturing, and the 18th century mines of Cornwall. His finding is that innovations from users are often more lasting and important than innovations from vendors. Hippel finds that while vendors focus on incremental improvements to known capabilities, users are more likely to come up with new capabilities that have important future uses.

Hippel focuses on “lead users” – users with extreme needs that have not yet been encountered by most other users. A lead user might be a surgeon practicing a new technique, a systems administrator under persistent network attack, or a mountain biker exploring new terrain. Lead users are motivated to figure out how to build new things with existing products. And, this book demonstrates, they often have the ability to make substantial progress. A weekend kiteboarder might be an aerospace engineer during the week. A surgeon might be able to raise funding to manufacture new equipment. An electronics manufacturer certainly has the means to make extensive customizations to the equipment it uses on its primary assembly line.

In contrast, vendors doing market research for a new product typically focus on their mainstream users. They make improvements in the speed, size, convenience, availability, usability, and reliability of their products, as typically used. Each iteration of the vendor product is produced largely by the same engineers building the same product for the same customers, but in an improved version. That’s the groove or rut that I see most software companies climbing into.

Vendors also segment their users into just a few categories, hoping to simplify their product planning and production by making a high-quality, one-size-fits-all product. In doing so, they miss what Hippel calls “heterogeneity of need”. Users of a product often have many different needs. In one survey of Apache Web server administrators, Hippel found that there were 92 different security related functions requested, and that there was no clustering or pattern in their requests. Some of those requests come from lead users, but they are hard to identify in the vendors aggregate view.

The most eye-popping statistics in this book come from a study of product development at 3M, where certain product development teams were coached to use a “lead user” method for identifying new product opportunities. They looked at emerging market trends, and then they identified users that were at the leading edge of those trends (the lead users) and went to them for ideas. A different set of product development teams used traditional market research and R&D to design their products. At the end of this study, the traditional teams had generated 41 incremental improvements and 1 major new product line, with forecast annual sales of $18M per product. The “lead user” teams had generated zero incremental improvements and five major new product lines, with forecast annual sales of $146M per product.

The software business has both the motivation and the ability to advance with user innovation. Heterogeneity of need is so high for large enterprise systems that virtually every installation is customized. User communities have leading-edge collaboration techniques, and communities have substantial programming resources that often include vendor capabilities.

Hippel gives us a handy formula for figuring out the build/buy decision. Assuming that either the vendor and the user can build a product or innovation for the same cost, a vendor spending 50% of non-sales expenses on R&D, with a 25% cost of sales, will need to sell 3 copies of a system before making any money. If the market is smaller, users will need to pick up the development task. At a 75% cost of sales, the vendor needs to sell 9 copies. This calculation, along with the variation in needs, starts to explain why, as the cost of selling a new software license has risen to over 75% of revenue, users have shifted resources from packaged systems to building custom systems with offshore and open source building blocks. This shifts resources toward user innovation.

In this situation, software producers must learn to take advantage of user innovation. They can provide toolkits to users (for example, source code), so that users can add more value through innovation. They can share the work of innovation. They can create processes for bringing user modifications back into the mainline product. In short, they can be more like open source communities.

User innovation works well, but is it in the interests of users or vendors to participate in an innovation community? Why reveal and contribute code if it just helps the free riders who don’t contribute? In the open source world, there may be tens of thousands of users for every contributor. What explains the increasing volume of commercial open source contributions? Hippel approaches this question by observing that the “free revealing” of important commercial innovation is widespread in many industries. He cites examples ranging from 18th century mining engineers who started a journal to share advances in steam engine technology, to IBM, which “after a lag” recently revealed its important copper interconnect chipmaking technology to competitors. In most of these cases, free revealing “is the only practical option.” Competitors can reproduce an innovation if they observe it, and patent protection is weak in most fields. Furthermore, individuals (consultants and developers) often have a strong incentive to freely reveal innovations that advance their careers or their personal missions. Free revealing increases the pace of innovation and benefits the mission of the innovator. By controlling the timing and placement of freely revealed innovations, commercial organizations can build up their platforms, their industries, their customer and partner participation, and other types of competitive advantage.

Hippel concludes by observing that user innovation and free revealing lead to enhanced “social welfare”. It can make everyone better off. User innovation is a powerful force that can benefit both users and vendors. This book helps you understand how it works so that you can make it work for you.

This article also appeared in IT Managers Journal.