Sunday, October 19, 2008

Why the Bugathon is OK, but does not solve the real problem

The numbers

Gerv did revive the Bugathon. So I went trough the bugs at layout tables to see how many bugs really need a testcase.

I did this in several steps.

  1. search for all bugs that do not contain the keyword testcase but do have a attachment which is named testcase, and assign the testcase keyword.

  2. search for all remaining bugs that do not contain the keyword testcase but do have a attachment which is of type text/html, decide if they are real testcases and assign the keyword

  3. Go through the remaining bugs, that do not contain the keyword testcase. During this step a lot of old unconfirmed showed up that have been WorksForMe. I know from previous experience that WFM's can be for ever not closed, so I closed them aggressively and asked to reopen with a clear explanation whats going on.

  4. I found 6 bugs where a testcase would be beneficial. There are 50 bugs that do not need a testcase but I did not mark them.
  5. So out of 450 bugs only 6 bugs need a testcase.

As a result we have now 400 table bugs with a testcase that are also marked by the keyword testcase.

In the last year 180 bugs got marked fixed in the layout-tables component. 36 of them have been assigned to nobody@mozilla.org and are fixed by other bugs.

In the same time period 136 bugs have been marked as WFM. This means we have roughly for every really fixed bug one WFM.

This hints that one needs to deal with the WFM category.

The WFM's

A triaged bug that ends purely as a WFM is a waste of resources. We waste

So the minimum requirement is that the testcase ends as testcase in our tree.

Yes, wfm too. Could you please flag the bugs you resolve with in-testsuite?
when there is testcase attached to the bug? I think it would be valuable to
add these as reftests because they are often edge cases and it makes sure we
have good code coverage in the tests. (Or even better, make a reftest and
just check it in ;-) )
Mats Palmgren


Converting usual teastcases to reftests requires knowledge and time. Its far more efficient to create a reftest when the bug is still present then to create one once the bug got WFM. Currently this is mainly done by the developers and this is where we really waste the most expensive resource that we have.

It would already help tremendously if a testcase would contain the bug and the expected rendering so that files needs only be split. This can be done by nearly everyone who is able to create a testcase.

Such a testcase has the obvious benefit that it can be easily converted into a reftest and it has also the advantage that triaging if the bug is still present becomes very efficient.

Continuous triage

I have seen a lot of bugs that had testcases that linger for years in our database. So if we really care about developer time then it would be far more efficient
to triage the existing bugs properly, to verify that they still exist and to make them compatible to the automated test frameworks rather than trying to get the latest bug out of firefox:general.
This triage requires experience that one only gets trough focus. This means experienced QA folks should take responsibility for fixed components and improve the bug quality in this components in a way that results in checkins into the tree.

Just by looking at the numbers of open bugs one should ask himself is this testcase self explaining in two, three or four years.
Can it be easily judged, that it is still an open bug. And this late triaging is the point where simply reviving the old Bugathon is not good enough. It is typically very difficult to see if a testcases that is WFM is really WFM.


The question for me looks like do we care for 6 or for 400 bugs. I would vote for the 400. And having on those 400 bugs reftests would help much more than the testcases on the 6 bugs

This page is powered by Blogger. Isn't yours?