« LINQ in PythonMemcached set invalidation »

Xtremely Quick Development System

02/18/08

Permalink 03:49:54 pm, by fumanchu Email , 1195 words   English (US)
Categories: IT

Xtremely Quick Development System

Divmod has a development methodology which they call UQDS. It's billed as lightweight, but I've been using it for 6 months now and find it burdensome. The basic flow of UQDS is: make a ticket, do all work in a branch, get a full review, merge to trunk. In theory, this brings the benefits of peer review and fewer conflicts between developers. In practice, however, I've found the following problems:

  1. Review is great, but is often performed by whomever is "free" rather than by those whom the change most affects. The larger the group affected, the more review happens after the change is merged back to trunk anyway.
  2. Conflicts are not reduced, they're delayed and distributed. One branch still gets merged to trunk before others, and those others often must all forward merge.
  3. The amount of change in a branch grows too large, and often incorporates changes which have little or nothing to do with the original ticket. Comments and whitespace get touched up from one dev to another; small buglets get found and fixed; refactoring happens. Review is more difficult.
  4. Commit history is unreadable. Changesets consist entirely of massive merges from branches and equally massive forward merges.
  5. The review overhead isn't worth the supposed benefits; developers spend too much time in review, and often invent bikeshed problems in order to feel their review time is worthwhile. There is no distinction between the review process for tiny versus massive changesets.
  6. Many problems still leak through the review process, but reverting trunk changesets is harder because they tend to be too large and mix concerns. That is, you often end up reverting 1/2 a changeset, which subversion does not make easy. UQDS tries to avoid reverting multiple changesets at once, but overcorrects by making fractional reverts more common. With Subversion, at least, I'd rather be saddled with the former.
  7. UQDS says, "[developers] can take all the time they need to come up with a good message" when they commit to trunk. In reality, they don't--they just want to finish quickly and move on to the next ticket.

So, here's my answer:

The Xtremely Quick Development System (XQDS)

The goals of XQDS:

  1. Code fast.
  2. Improve the process documentation.

The strategy of XQDS:

  1. Reduce changeset size. This allows everyone to code faster, since they don't need to exert as much mental effort to understand others' changes. It also makes the timeline more readable (and revertible!), since each changeset and its commit message is atomic.
  2. Resolve conflicts immediately.
  3. Apply review resources only where needed.
  4. Make tests easier to run than not run. If your test suite takes so long that you're forced to run it on a remote machine, you've either done something wrong or this methodology is too small for you. Which is fine.
  5. Record all decisions: on tickets where warranted, on trunk changesets regardless.

The flow of XQDS is:

  1. A task is created in an issue tracker.
  2. The task is discussed, on the ticket if possible. If the conversation is long or ephemeral, it may be conducted on IRC or elsewhere; however, at least the final decision(s) should be summarized on the ticket, not on a wiki page, mailing list, or other medium. This step may be ongoing as work is done.
  3. Someone accepts the ticket. If work lapses, others are always free to accept it for themselves.
  4. The worker does work in their local copy of trunk. Branches are created sparingly as needed, and the worker switches their local copy using svn switch.
  5. Work gets merged to trunk as it passes the full test suite in the smallest functional chunks possible. Sometimes "smallest possible" can actually be quite large; however, refactorings, buglets, and doc improvements are committed in their own changesets. In my experience, there's nothing worse than trying to review a changeset that's 5% ticket fix and 95% whitespace changes.
  6. Each commit includes a message that MUST describe the actual change; comments about the reasons or context are desirable but secondary. However, commits that directly influence a ticket always reference the ticket.
  7. Everyone runs svn up on at least a daily basis, and definitely before committing. Conflicts are resolved as needed with each local copy.

Questions:

  • Don't you lose the benefit of branching? What if Jethro checks in broken code and goes to lunch?

    1. Jethro shouldn't be checking in broken code. You fix this by increasing peer shame, not ignoring it or shifting the detection work onto another developer.
    2. This happens even using UQDS. Review by a single developer doesn't catch all possible broken code.
    3. You can still branch whenever you see fit. You're just not required to branch for each ticket.
  • Don't you lose the benefit of review? Review helps avoid conflict and also teaches the reviewee.

    1. In my experience, review is better with XQDS than UQDS. First, you apply review resources where they are needed. Not every change needs review. This allows developers to invest more energy into reviews which merit it.
    2. You raise the bar for review: everyone is expected to watch the timeline, svn up often, and help resolve conflicts.
    3. Senior developers are allowed to touch up junior developers' code in situ. It's always faster and often much more effective to show improvements than explain them.
    4. People who care about or are particularly skilled about various aspects of code quality are themselves responsible for upkeep. At some point, you can stop trying to teach Jethro how to avoid contention over concurrent shared mutable resources because he's never going to get it. At some point, you stop trying to make the Twisted developer follow your PEP-8 whitespace conventions because she has no incentive to do so; you fix it yourself and get on with life.
  • What if I need to share unfinished code with another developer? Or switch developers mid-feature? Or switch platforms mid-feature? Or switch features mid-developer?

    • Make a branch. XQDS doesn't prohibit this. But it doesn't mandate it like UQDS does. All that Combinator nonsense can be replaced with svn switch and a simple folder rename when you want to put aside some work for a while.
  • Isn't trunk broken more often?

    • Not if you have a good local test suite. If your test suite can't detect broken code, you need to write more functional (end-to-end) tests.
  • UQDS says it improves information flow to managers. Don't you lose that with XQDS?

    • Not at all. On the one hand, managers use the same tools (e.g. Trac timelines) that developers use to monitor progress. But with XQDS, the timeline is actually readable without all those cross merges. Managers can therefore also write Trac SQL queries, for example, which report developer activity based on the changeset activity.
  • Doesn't XQDS require more conflict resolution?

    • Not more, just more immediate. As UQDS notes, this can result in a situation where you feel like you cannot commit your local work because it's now broken due to someone else's changes. But that's a false impression: just svn switch to a new branch and commit your (now broken) changes. You're going to have to resolve the conflict either way; but XQDS allows you to skip making a branch unless you need it.

7 comments

Comment from: mike bayer [Visitor]

"XQDS" is what pretty much most of us in opensource-land are doing anyway. Must it have a name ? I'm just going to forget it and then be accused of being non-XQDS compliant. Maybe call it "TOV": Timeline Oriented Development. Since a lot of the advantages you name above have to do with the path of code sticking mostly to a singular, nicely documented chronology which is easily viewable as a timeline.

02/18/08 @ 18:05
Comment from: fumanchu [Member] Email

Mike,

Like all things, giving it a name makes referring to it easier. I'd be just as happy with "TOV". You're right about the timeline; it seems to be one of the bigger distinctions: UQDS tries to pretty up revision logs, XQDS tries to pretty up the Trac timeline. The other big difference is when/whether to branch, so maybe we should call it "TTOV" for "Trunk/Timeline Oriented Development". :P

02/18/08 @ 19:49
Comment from: Manuzhai [Visitor]

You should check out something more modern than SVN. Using Mercurial (or git, or bzr) at the very least solves the problem of merges being one big changeset all squashed together.

02/19/08 @ 01:07
Comment from: Kent Johnson [Visitor] · http://kentsjohnson.com

XQDS is pretty much what we did at my last job. It worked fine with a team of about a half-dozen developers. We only made branches for maintenance and for substantial changes that we knew would take a long time to be ready.

02/19/08 @ 06:34
Comment from: Lakin [Visitor] · http://lakin.weckers.net

To be fair, I didn't think that UQDS claimed that conflicts would be fewer. Rather just that they would happen when one is in a 'checkin' or 'conflict-resolution' type mindset, rather than in the midst of solving a problem.

On the other hand I also agree that UQDS has been burdensome for small tickets and recently many of us (ssshhh! don't tell the boss) haven't been using it as much for those types of tickets. When we get large tickets where we'd like to be able to share (potentially broken) code between developers we make branches.


I still find Combinator useful for maintaining branches and pathing between branches, it saves me having to remember and chain together multiple svn commands with revision arguments.

So in summary, I'm all for adopting a "branch only when you need it" methodology, but I won't be giving up combinator yet. :P

02/19/08 @ 08:24
Comment from: Marius Gedminas [Visitor] · http://gedmin.as

I've apparently been using XQDS at work for the last several years. I've been looking forward to maybe one day switching to Bazaar or GIT (from Subversion) and requiring branch reviews before merging to trunk. Your post made me think whether that would really be such a good idea.

02/19/08 @ 11:04
Comment from: Glyph Lefkowitz [Visitor] Email · http://glyph.twistedmatrix.com/

Hi Bob,

I realize it's been quite a while since you wrote this post, but I'm just noticing it now; I hope you don't mind if I correct some misconceptions about UQDS here.

I'll respond to each of your seven points:

1.: UQDS doesn't have much to say about how the determination is made who to assign a review branch to. On the Twisted project, we assign it to nobody, and the first available reviewer picks it up. On Divmod projects, the author selects someone based on their preferences for that branch. If you are finding that reviews are not assigned to appropriate people, then determine an algorithm that selects an appropriate person!

2.: Conflict reduction isn't really a goal of using branches. The only reference to reducing conflicts in the UQDS writeup is the part that says "branch lifetimes should be short".

3.: Keeping developers on-task is a delicate art, but UQDS specifies that the branch for a ticket should be as small as possible to satisfy that ticket. I don't know why you were having this problem, but it certainly doesn't have anything to do with the process itself: the process says not to do it. If anything, UQDS gives you a specific way to say "if you go off task, put other stuff somewhere else, don't pollute this changeset."

4.: This is an issue with subversion (if it's an issue at all), not UQDS. You can do UQDS with bzr, and then you get a much richer commit history. At any rate, there's hardly any point in reading the commit logs at /, you should be reading /trunk; then you only see merges.

5.: I suppose whether the review overhead is worth the benefits is ultimately a matter of taste, but developers should not be inventing bikeshed problems. To be fair, the summary / introduction on the UQDS page is really vague about what the reviewer is supposed to be doing, but later it is much more specific: "The reviewer verifies that the code makes sense, has unit tests, has documentation, updates any existing documentation that is affected, and actually resolves the issue in question." The reviewer needs to avoid involving personal judgments of taste as required review criteria. Training people to do good code reviews is not easy, but the answer is not (as you posit in XQDS) getting them to do fewer reviews.

6.: Let me make something very clear. PARTIAL REVERTS ARE NEVER, EVER, EVER, EVER, EVER, EVER, EVER allowed in UQDS. Not EVER. This is possibly the worst thing you can do. The idea is, trunk is a queue of changes which are made. If one change is bad (i.e. causes unit tests to fail), it gets reversed; automatically, completely, and without prejudice or ego getting involved. If you start involving random cherry-picking on trunk, then you've created an entirely new unverified change. If you know that part of the change is good, then create a new branch, shepherd the thought-to-be-good part of the change through the review process, and make sure that it fully satisfies some requirement.

Point 6 is absolutely the worst thing about this post, since it suggests that what you're actually criticizing here is actually the opposite of UQDS.

For emphasis: partial reverts are not part of UQDS. They are never allowed.

7.: This is actually a pretty good point. Given that the purpose of the branch should be decided upon ahead of time, the commit message should also be written ahead of time. The reviewer should review the commit message along with everything else. We've been talking a lot lately about improving the UQDS spec to deal with this in a way similar to the way that the Bazaar developers do code reviews.

Anyway, thanks for your feedback and I'm sorry I didn't get to it sooner!

06/24/08 @ 17:41

Leave a comment


Your email address will not be revealed on this site.

Your URL will be displayed.

Please enter the phrase "I am a real human." in the textbox above.
(Line breaks become <br />)
(Name, email & website)
(Allow users to contact you through a message form (your email will not be revealed.)
July 2016
Sun Mon Tue Wed Thu Fri Sat
 << <   > >>
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31            

Search

The requested Blog doesn't exist any more!

XML Feeds

powered by b2evolution free blog software