Summary
This document describes our experiment data for case studies of six real-world multilingual bugs in the following paper.
Contents
- Target source code, bug report, bug fix
- Target source code : original target source code obtained from public URL
- Bug report : A bug description reported by a developer or a researcher
- Bug fix : A code change that fixes the reported bug
- Testing object, test case
- Testing object : A modified target source code
- Test case : A list of test cases used on our experiment
- Mutant generation
- Mutant : # of mutants (total, generated by existing mutation operator, generated by new mutation operator)
- Mutation operator : # of mutation operator (total, existing, new)
- Experiment result
- Suspiciousness score, ranking : suspiciousness score and ranking of MUSEUM, Jaccard, Ochiai, Op2 for each code line
- Mutation testing result : A list of mutation testing result for each mutant
- Line : mutated line of testing object
- LineCovTests : # of test cases that executes a line
- Pass : # of passing test cases
- Fail : # of failing test cases
- Fail2Pass : # of test cases that fails on original program but passes on the mutant
- Pass2Fail : # of test cases that passes on original program but fails on the mutant
- Mutants : A list of generated mutants
An overview of experiment results
The ranks of the buggy line identified by MUSEUM and other SBFL techniques
|
Bug1 |
Bug2 |
Bug3 |
Bug4 |
Bug5 |
Bug6 |
Bug7 |
Bug8 |
MUSEUM |
1 (0.1%) |
2 (0.7%) |
1 (0.2%) |
1 (0.1%) |
8 (4.3%) |
3 (0.2%) |
1 (0.2%) |
1 (0.02%) |
Jaccard |
80 (4.1%) |
4 (1.3%) |
5 (1.1%) |
83 (44.6%) |
61 (32.8%) |
3494 (100%) |
84 (17.5%) |
574 (10.2%) |
Ochiai |
80 (4.1%) |
4 (1.3%) |
5 (1.1%) |
83 (44.6%) |
61 (32.8%) |
3494 (100%) |
84 (17.5%) |
574 (10.2%) |
Op2 |
80 (4.1%) |
4 (1.3%) |
5 (1.1%) |
83 (44.6%) |
61 (32.8%) |
3494 (100%) |
84 (17.5%) |
574 (10.2%) |
Overview of the experiment data
|
Bug1 |
Bug2 |
Bug3 |
Bug4 |
Bug5 |
Bug6 |
Bug7 |
Bug8 |
# of test cases used |
8 |
150 |
159 |
170 |
184 |
50 |
112 |
204 |
# of the target lines (executed by the failing test case) |
1939 |
299 |
443 |
186 |
186 |
3494 |
294 |
4998 |
# of mutants |
2861 |
691 |
965 |
718 |
369 |
9479 |
844 |
14490 |
# of lines which have a mutant |
1575 |
219 |
327 |
132 |
103 |
2524 |
226 |
3855 |
# of mutants that make a passing test case fails (breaking) |
305 |
462 |
681 |
364 |
311 |
3044 |
542 |
8766 |
# of mutants that make a failing test case passes (partial fix) |
1 |
3 |
7 |
2 |
51 |
32 |
3 |
1 |
Time cost (min) |
12 |
60 |
45 |
25 |
23 |
175 |
50 |
511 |
Target source code, bug report, bug fix
Target source code
Bug report
Bug fix
Diff between rev. 1.63 and rev. 1.64 |
=== modified file '/cvsroot/azureus/azureus3/com/aelitis/azureus/ui/swt/views/list/ListView.java' |
--- ListView.java 15 Dec 2007 00:46:23 -0000 1.63 |
+++ ListView.java 12 Jan 2008 17:35:13 -0000 1.64 |
    public Listview ... { |
      ... |
      private Image imgView = null; |
      ... |
      public void handleResize(boolean bForce){ |
        ... |
        if(imgView == null || bForce){ |
          ... |
+         if (imgView != null && !imgView.isDisposed()) |
+           imgView.dispose(); |
          imgView = new Image(...); |
          ... |
|
Testing object, test case
Testing object
|
Size of target program |
Java |
Native C |
Files |
LOC |
Files |
LOC |
bug1 |
2705 |
340.6K |
N/A |
N/A |
Test case
|
Total |
Used |
Pass |
Fail |
# of test cases |
8 |
8 |
7 |
1 |
We created unit test cases of the ListView class of Azureus.
- The failing test case is constructed based on the error scenario described in the QVM paper.
- 1st column : Test case - Name of test case
- 2nd column : Test result - 0(fail) or 1(pass)
- 3rd column : Source file - Location of source file containing test code
Mutant generation
Mutant
|
Total |
generated by existing mutation operators |
generated by new mutation operators |
# of mutants |
2861 |
2710 |
151 |
Mutation operator
|
Total |
Existing |
New |
# of mutation operators |
14 |
13 |
1 |
- 1st column : Mutant ID - Unique ID for mutant
- 2nd column : Mutated Line - Mutated line of testing object
- 3rd column : Mutation Operator - Mutation operator applied to the mutated line
Experiment result
Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)
- 1st column : Line - Line of testing object
- 2nd column : MuseSusp - Suspiciousness score of MUSEUM
- 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
- 4th column : JaccardSusp - Suspiciousness score of Jaccard
- 5th column : JaccardRank - Suspiciousness ranking of Jaccard
- 6th column : OchiaiSusp - Suspiciousness score of Ochiai
- 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
- 8th column : Op2Susp - Suspiciousness score of Op2
- 9th column : Op2Rank - Suspiciousness ranking of Op2
Mutation testing result
|
make a passing test case fails (breaking) |
make a failing test case passes (partial fix) |
golden mutants (complete fix) |
# of mutants |
305 |
1 |
1 |
- 1st column : Line - Mutated line of testing object
- 2nd column : LineCovTests - # of test cases that covers the mutated line
- 3rd column : Pass - # of passing test cases in LineCovTests
- 4th column : Fail - # of failing test cases in LineCovTests
- 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
- 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
- 7th column : Mutants - a list of generated mutants
Target source code, bug report, bug fix
Target source code
Bug report
Bug fix
The bug is fixed by the developer at the following commit
Diff between 1568679 and 8e825f7 |
=== modified file 'src/main/java/org/sqlite/Stmt.java' |
    public int executedUpdate(String sql){ |
      ... |
+     changes = db.total_changes(); |
      ... |
      db._exec(sql); |
-     changes = db.changes(); |
+     changes = db.total_changes() - changes; |
      ... |
      return changes; |
    } |
|
Testing object, test case
Testing object
|
Size of target program |
Java |
Native C |
Files |
LOC |
Files |
LOC |
bug2 |
20 |
4.6K |
3 |
1.8K |
Test case
|
Total |
Used |
Pass |
Fail |
# of test cases |
151 |
150 |
149 |
1 |
1 test case is excluded because the test case contains empty body.
We created one failing test case based on the bug report and the existing test cases.
- The failing test case has an assertion that the return value of executeUpdate() with a DDL query must be zero.
- Although the bug report provides a new test case, we cannot use the test case because it does not properly check that executeUpdate() works correctly with a DDL query.
- 1st column : Test case - Name of test case
- 2nd column : Test result - 0(fail) or 1(pass)
- 3rd column : Source file - Location of source file containing test code
Mutant generation
Mutant
|
Total |
generated by existing mutation operators |
generated by new mutation operators |
# of mutants |
691 |
603 |
88 |
Mutation operator
|
Total |
Existing |
New |
# of mutation operators |
49 |
37 |
12 |
- 1st column : Mutant ID - Unique ID for mutant
- 2nd column : Mutated Line - Mutated line of testing object
- 3rd column : Mutation Operator - Mutation operator applied to the mutated line
Experiment result
Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)
- 1st column : Line - Line of testing object
- 2nd column : MuseSusp - Suspiciousness score of MUSEUM
- 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
- 4th column : JaccardSusp - Suspiciousness score of Jaccard
- 5th column : JaccardRank - Suspiciousness ranking of Jaccard
- 6th column : OchiaiSusp - Suspiciousness score of Ochiai
- 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
- 8th column : Op2Susp - Suspiciousness score of Op2
- 9th column : Op2Rank - Suspiciousness ranking of Op2
Mutation testing result
|
make a passing test case fails (breaking) |
make a failing test case passes (partial fix) |
golden mutants (complete fix) |
# of mutants |
555 |
3 |
0 |
- 1st column : Line - Mutated line of testing object
- 2nd column : LineCovTests - # of test cases that covers the mutated line
- 3rd column : Pass - # of passing test cases in LineCovTests
- 4th column : Fail - # of failing test cases in LineCovTests
- 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
- 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
- 7th column : Mutants - a list of generated mutants
Target source code, bug report, bug fix
Target source code
Bug report
Bug fix
The bug is fixed by the developer at the following commit
Diff between 5a06a06 and 9185323 |
=== modified file 'src/main/java/org/sqlite/PrepStmt.java' |
   final class PrepStmt { |
     ... |
     public void clearParameters() throws SQLException { |
       checkOpen(); |
-      db.reset(pointer); |
-      clearBatch(); |
+      db.clear_bindings(pointer); |
+      batch = null; |
     } |
   } |
|
Testing object, test case
Testing object
|
Size of target program |
Java |
Native C |
Files |
LOC |
Files |
LOC |
bug3 |
19 |
4.2K |
2 |
1.7K |
Test case
|
Total |
Used |
Pass |
Fail |
# of test cases |
159 |
159 |
158 |
1 |
1 test case is excluded because the test case contains empty body.
- 1st column : Test case - Name of test case
- 2nd column : Test result - 0(fail) or 1(pass)
- 3rd column : Source file - Location of source file containing test code
Mutant generation
Mutant
|
Total |
generated by existing mutation operators |
generated by new mutation operators |
# of mutants |
965 |
867 |
98 |
Mutation operator
|
Total |
Existing |
New |
# of mutation operators |
53 |
41 |
12 |
- 1st column : Mutant ID - Unique ID for mutant
- 2nd column : Mutated Line - Mutated line of testing object
- 3rd column : Mutation Operator - Mutation operator applied to the mutated line
Experiment result
Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)
- 1st column : Line - Line of testing object
- 2nd column : MuseSusp - Suspiciousness score of MUSEUM
- 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
- 4th column : JaccardSusp - Suspiciousness score of Jaccard
- 5th column : JaccardRank - Suspiciousness ranking of Jaccard
- 6th column : OchiaiSusp - Suspiciousness score of Ochiai
- 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
- 8th column : Op2Susp - Suspiciousness score of Op2
- 9th column : Op2Rank - Suspiciousness ranking of Op2
Mutation testing result
|
make a passing test case fails (breaking) |
make a failing test case passes (partial fix) |
golden mutants (complete fix) |
# of mutants |
793 |
7 |
1 |
- 1st column : Line - Mutated line of testing object
- 2nd column : LineCovTests - # of test cases that covers the mutated line
- 3rd column : Pass - # of passing test cases in LineCovTests
- 4th column : Fail - # of failing test cases in LineCovTests
- 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
- 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
- 7th column : Mutants - a list of generated mutants
Target source code, bug report, bug fix
Target source code
Bug report
Bug fix
The reporter suggested a quick fix at the bug report.
>>> Fix <<<
A quick fix is to make the "receiver" global reference.
src/jni/bindings_java_signal.c:
  524 bjc->receiver = (*env)->NewGlobalRef(env,receiver);
The fix has not been confirmed by developer.
Testing object, test case
Testing object
|
Size of target program |
Java |
Native C |
Files |
LOC |
Files |
LOC |
bug4 |
1097 |
64.2K |
496 |
65.6K |
Test case
|
Total |
Used |
Pass |
Fail |
# of test cases |
176 |
170 |
169 |
1 |
We used 1 failing test case from 7 failing test cases.
- The one that used for the fist error-reproducing test case in the bug report
- All 7 failing test cases are infected by the same bug (not the same errors) because the same fix resolve all errors
- 1st column : Test case - Name of test case
- 2nd column : Test result - 0(fail) or 1(pass)
- 3rd column : Source file - Location of source file containing test code
Mutant generation
Mutant
|
Total |
generated by existing mutation operators |
generated by new mutation operators |
# of mutants |
718 |
677 |
41 |
Mutation operator
|
Total |
Existing |
New |
# of mutation operators |
47 |
37 |
10 |
|
- 1st column : Mutant ID - Unique ID for mutant
- 2nd column : Mutated Line - Mutated line of testing object
- 3rd column : Mutation Operator - Mutation operator applied to the mutated line
Experiment result
Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)
- 1st column : Line - Line of testing object
- 2nd column : MuseSusp - Suspiciousness score of MUSEUM
- 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
- 4th column : JaccardSusp - Suspiciousness score of Jaccard
- 5th column : JaccardRank - Suspiciousness ranking of Jaccard
- 6th column : OchiaiSusp - Suspiciousness score of Ochiai
- 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
- 8th column : Op2Susp - Suspiciousness score of Op2
- 9th column : Op2Rank - Suspiciousness ranking of Op2
Mutation testing result
|
make a passing test case fails (breaking) |
make a failing test case passes (partial fix) |
golden mutants (complete fix) |
# of mutants |
358 |
2 |
2 |
- 1st column : Line - Mutated line of testing object
- 2nd column : LineCovTests - # of test cases that covers the mutated line
- 3rd column : Pass - # of passing test cases in LineCovTests
- 4th column : Fail - # of failing test cases in LineCovTests
- 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
- 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
- 7th column : Mutants - a list of generated mutants
Target source code, bug report, bug fix
Target source code
Bug report
Commit log
Contributor |
Andrew Cowie |
Date |
07-June-2009 at 09:54 |
Commit Comment |
Avoid segfault lurking in GtkSpell library
It seems the implementation in the GtkSpell library has a signal
handler connected to the GtkTextView 'destroy' signal calling its
internal gtkspell_free() function. If we call gtkspell_detach() it
results in a segmentation fault.
|
Bug fix
The bug is fixed by the developer at revision 659
Diff between rev. 658 and rev. 659 |
=== modified file 'src/bindings/org/gnome/gtk/Spell.java' |
--- src/bindings/org/gnome/gtk/Spell.java 2009-06-07 11:00:30 +0000 |
+++ src/bindings/org/gnome/gtk/Spell.java 2009-06-07 13:54:01 +0000 |
    protected void release() { |
-      GtkSpell.detach(this); |
+   // GtkSpell.detach(this); |
    } |
|
Testing object, test case
Testing object
|
Size of target program |
Java |
Native C |
Files |
LOC |
Files |
LOC |
bug5 |
1134 |
67.1K |
514 |
69.2K |
Test case
|
Total |
Used |
Pass |
Fail |
# of test cases |
194 |
184 |
183 |
1 |
10 test cases are excluded because of the test cases are duplicated
We used 1 failing test cases and 183 passing test cases
- 1 failing test case: we wrote a failing test case based on the bug report and the existing test cases since no error-reproducing test case is given.
- 183 passing test case: we used the regression test cases in the Java-gnome repository (revision 659) for passing test case executions.
- 1st column : Test case - Name of test case
- 2nd column : Test result - 0(fail) or 1(pass)
- 3rd column : Source file - Location of source file containing test code
Mutant generation
Mutant
|
Total |
generated by existing mutation operators |
generated by new mutation operators |
# of mutants |
369 |
340 |
29 |
Mutation operator
|
Total |
Existing |
New |
# of mutation operators |
45 |
35 |
10 |
- 1st column : Mutant ID - Unique ID for mutant
- 2nd column : Mutated Line - Mutated line of testing object
- 3rd column : Mutation Operator - Mutation operator applied to the mutated line
Experiment result
Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)
- 1st column : Line - Line of testing object
- 2nd column : MuseSusp - Suspiciousness score of MUSEUM
- 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
- 4th column : JaccardSusp - Suspiciousness score of Jaccard
- 5th column : JaccardRank - Suspiciousness ranking of Jaccard
- 6th column : OchiaiSusp - Suspiciousness score of Ochiai
- 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
- 8th column : Op2Susp - Suspiciousness score of Op2
- 9th column : Op2Rank - Suspiciousness ranking of Op2
Mutation testing result
|
make a passing test case fails (breaking) |
make a failing test case passes (partial fix) |
golden mutants (complete fix) |
# of mutants |
311 |
51 |
15 |
- 1st column : Line - Mutated line of testing object
- 2nd column : LineCovTests - # of test cases that covers the mutated line
- 3rd column : Pass - # of passing test cases in LineCovTests
- 4th column : Fail - # of failing test cases in LineCovTests
- 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
- 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
- 7th column : Mutants - a list of generated mutants
Target source code, bug report, bug fix
Target source code
Target version is SWT-3.7.0.3
Bug report
The bug is first reported at SWT-3.5.0 by the developers
Bug fix
The patch is applied for SWT-3.7.0.4 by the developers
Diff between commit 5b6a9a and commit 7f81e1 |
=== modified file 'gtk/org/eclipse/swt/widgets/Display.java' |
   public class Display { |
     ... |
     void initializeSubclasses() { |
       if (OS.GTK_VERSION >= OS.VERSION (2, 4, 0)) { |
         OS.G_OBJECT_CLASS_CONSTRUCTOR(PLClass); |
         OS.G_OBJECT_CLASS_SET_CONSTRUCTOR(PLClass, NewProc); |
         p = OS.G_OBJECT_CLASS_CONSTRUCTOR(PLClass); |
         OS.G_OBJECT_CLASS_SET_CONSTRUCTORR(PLClass, new NewProcCB(p)); |
     ... |
|
Testing object, test case
Testing object
|
Size of target program |
Java |
Native C |
Files |
LOC |
Files |
LOC |
bug6 |
582 |
118.7K |
29 |
20.7K |
Test case
|
Total |
Used |
Pass |
Fail |
# of test cases |
50 |
50 |
49 |
1 |
- We used total 49 test cases with 1 failing test case
- 1 failing test case: the test case was given with a bug report, and used for the Blink paper.
48 passing test cases: we chose the test cases for the Display module only because all error-stack traces in the duplicated bug reports commonly have at least one Display method.
- 1st column : Test case - Name of test case
- 2nd column : Test result - 0(fail) or 1(pass)
- 3rd column : Source file - Location of source file containing test code
Mutant generation
Mutant
|
Total |
generated by existing mutation operators |
generated by new mutation operators |
# of mutants |
9479 |
8926 |
553 |
Mutation operator
|
Total |
Existing |
New |
# of mutation operators |
79 |
66 |
13 |
- 1st column : Mutant ID - Unique ID for mutant
- 2nd column : Mutated Line - Mutated line of testing object
- 3rd column : Mutation Operator - Mutation operator applied to the mutated line
Experiment result
Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)
- 1st column : Line - Line of testing object
- 2nd column : MuseSusp - Suspiciousness score of MUSEUM
- 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
- 4th column : JaccardSusp - Suspiciousness score of Jaccard
- 5th column : JaccardRank - Suspiciousness ranking of Jaccard
- 6th column : OchiaiSusp - Suspiciousness score of Ochiai
- 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
- 8th column : Op2Susp - Suspiciousness score of Op2
- 9th column : Op2Rank - Suspiciousness ranking of Op2
Mutation testing result
|
make a passing test case fails (breaking) |
make a failing test case passes (partial fix) |
golden mutants (complete fix) |
# of mutants |
3612 |
17 |
14 |
- 1st column : Line - Mutated line of testing object
- 2nd column : LineCovTests - # of test cases that covers the mutated line
- 3rd column : Pass - # of passing test cases in LineCovTests
- 4th column : Fail - # of failing test cases in LineCovTests
- 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
- 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
- 7th column : Mutants - a list of generated mutants
Target source code, bug report, bug fix
Target source code
Bug report
Bug fix
The bug is fixed by the developer at the following commit
Diff between 847f364 and c4e0e7c |
=== modified file 'src/main/java/org/sqlite/NativeDB.c' |
   static xCall (...) { |
     ... |
+     // check if xFunc threw an Exception |
+     if (*env)->ExceptionCheck(env)) { |
+       xFunc_error(context, env); |
+     } |
     (*env)->SetLongField(env, func, fld_context, 0); |
     (*env)->SetLongField(env, func, fld_value, 0); |
     (*env)->SetIntField(env, func, fld_args, 0); |
-     // check if xFunc threw an Exception |
-     if ((*env)->ExceptionCheck(env)) xFunc_error(context, env); |
Testing object, test case
Testing object
|
Size of target program |
Java |
Native C |
Files |
LOC |
Files |
LOC |
bug3 |
25 |
4.9K |
2 |
0.6K |
Test case
|
Total |
Used |
Pass |
Fail |
# of test cases |
112 |
112 |
111 |
1 |
- 1st column : Test case - Name of test case
- 2nd column : Test result - 0(fail) or 1(pass)
- 3rd column : Source file - Location of source file containing test code
Mutant generation
Mutant
|
Total |
generated by existing mutation operators |
generated by new mutation operators |
# of mutants |
844 |
700 |
144 |
Mutation operator
|
Total |
Existing |
New |
# of mutation operators |
52 |
40 |
12 |
- 1st column : Mutant ID - Unique ID for mutant
- 2nd column : Mutated Line - Mutated line of testing object
- 3rd column : Mutation Operator - Mutation operator applied to the mutated line
Experiment result
Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)
- 1st column : Line - Line of testing object
- 2nd column : MuseSusp - Suspiciousness score of MUSEUM
- 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
- 4th column : JaccardSusp - Suspiciousness score of Jaccard
- 5th column : JaccardRank - Suspiciousness ranking of Jaccard
- 6th column : OchiaiSusp - Suspiciousness score of Ochiai
- 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
- 8th column : Op2Susp - Suspiciousness score of Op2
- 9th column : Op2Rank - Suspiciousness ranking of Op2
Mutation testing result
|
make a passing test case fails (breaking) |
make a failing test case passes (partial fix) |
golden mutants (complete fix) |
# of mutants |
542 |
3 |
3 |
- 1st column : Line - Mutated line of testing object
- 2nd column : LineCovTests - # of test cases that covers the mutated line
- 3rd column : Pass - # of passing test cases in LineCovTests
- 4th column : Fail - # of failing test cases in LineCovTests
- 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
- 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
- 7th column : Mutants - a list of generated mutants
Target source code, bug report, bug fix
Target source code
Target version is SWT-4.4
Bug report
The bug is first reported by the developers and has not resolved (open bug)
Testing object, test case
Testing object
|
Size of target program |
Java |
Native C |
Files |
LOC |
Files |
LOC |
bug8 |
591 |
126.6K |
29 |
48.5K |
Test case
|
Total |
Used |
Pass |
Fail |
# of test cases |
204 |
204 |
203 |
1 |
- 1st column : Test case - Name of test case
- 2nd column : Test result - 0(fail) or 1(pass)
- 3rd column : Source file - Location of source file containing test code
Mutant generation
Mutant
|
Total |
generated by existing mutation operators |
generated by new mutation operators |
# of mutants |
14490 |
13450 |
1040 |
Mutation operator
|
Total |
Existing |
New |
# of mutation operators |
59 |
43 |
16 |
- 1st column : Mutant ID - Unique ID for mutant
- 2nd column : Mutated Line - Mutated line of testing object
- 3rd column : Mutation Operator - Mutation operator applied to the mutated line
Experiment result
Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)
- 1st column : Line - Line of testing object
- 2nd column : MuseSusp - Suspiciousness score of MUSEUM
- 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
- 4th column : JaccardSusp - Suspiciousness score of Jaccard
- 5th column : JaccardRank - Suspiciousness ranking of Jaccard
- 6th column : OchiaiSusp - Suspiciousness score of Ochiai
- 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
- 8th column : Op2Susp - Suspiciousness score of Op2
- 9th column : Op2Rank - Suspiciousness ranking of Op2
Mutation testing result
|
make a passing test case fails (breaking) |
make a failing test case passes (partial fix) |
golden mutants (complete fix) |
# of mutants |
8766 |
1 |
1 |
- 1st column : Line - Mutated line of testing object
- 2nd column : LineCovTests - # of test cases that covers the mutated line
- 3rd column : Pass - # of passing test cases in LineCovTests
- 4th column : Fail - # of failing test cases in LineCovTests
- 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
- 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
- 7th column : Mutants - a list of generated mutants