MUSEUM: Mutation based Debugging for Real-world Multilingual Programs

Summary

This document describes our experiment data for case studies of six real-world multilingual bugs in the following paper.


Contents

  • Target source code, bug report, bug fix
    • Target source code : original target source code obtained from public URL
    • Bug report : A bug description reported by a developer or a researcher
    • Bug fix : A code change that fixes the reported bug
  • Testing object, test case
    • Testing object : A modified target source code
    • Test case : A list of test cases used on our experiment
  • Mutant generation
    • Mutant : # of mutants (total, generated by existing mutation operator, generated by new mutation operator)
    • Mutation operator : # of mutation operator (total, existing, new)
  • Experiment result
    • Suspiciousness score, ranking : suspiciousness score and ranking of MUSEUM, Jaccard, Ochiai, Op2 for each code line
    • Mutation testing result : A list of mutation testing result for each mutant
      • Line : mutated line of testing object
      • LineCovTests : # of test cases that executes a line
      • Pass : # of passing test cases
      • Fail : # of failing test cases
      • Fail2Pass : # of test cases that fails on original program but passes on the mutant
      • Pass2Fail : # of test cases that passes on original program but fails on the mutant
      • Mutants : A list of generated mutants

An overview of experiment results

The ranks of the buggy line identified by MUSEUM and other SBFL techniques

Bug1 Bug2 Bug3 Bug4 Bug5 Bug6 Bug7 Bug8
MUSEUM 1
(0.1%)		 2
(0.7%)		 1
(0.2%)		 1
(0.1%)		 8
(4.3%)		 3
(0.2%)		 1
(0.2%)		 1
(0.02%)
Jaccard 80
(4.1%)		 4
(1.3%)		 5
(1.1%)		 83
(44.6%)		 61
(32.8%)		 3494
(100%)		 84
(17.5%)		 574
(10.2%)
Ochiai 80
(4.1%)		 4
(1.3%)		 5
(1.1%)		 83
(44.6%)		 61
(32.8%)		 3494
(100%)		 84
(17.5%)		 574
(10.2%)
Op2 80
(4.1%)		 4
(1.3%)		 5
(1.1%)		 83
(44.6%)		 61
(32.8%)		 3494
(100%)		 84
(17.5%)		 574
(10.2%)

Overview of the experiment data

Bug1 Bug2 Bug3 Bug4 Bug5 Bug6 Bug7 Bug8
# of test cases used 8 150 159 170 184 50 112 204
# of the target lines
(executed by the
failing test case)		 1939 299 443 186 186 3494 294 4998
# of mutants 2861 691 965 718 369 9479 844 14490
# of lines which
have a mutant		 1575 219 327 132 103 2524 226 3855
# of mutants that
make a passing test
case fails (breaking)		 305 462 681 364 311 3044 542 8766
# of mutants that
make a failing test
case passes
(partial fix)		 1 3 7 2 51 32 3 1
Time cost (min) 12 60 45 25 23 175 50 511

Target source code, bug report, bug fix

Target source code

http://sourceforge.net/projects/azureus/files/x_obsolete/azureus-3.0.4.2/Azureus_3.0.4.2_linux.tar.bz2/download


Bug report

Matthew Arnold et. al. "QVM: An Efficient Runtime for Detecting Defects in Deployed Systems"


Bug fix

http://sourceforge.net/p/azureus/mailman/message/18318135/

Diff between
rev. 1.63
and
rev. 1.64
=== modified file '/cvsroot/azureus/azureus3/com/aelitis/azureus/ui/swt/views/list/ListView.java'
--- ListView.java 15 Dec 2007 00:46:23 -0000 1.63
+++ ListView.java 12 Jan 2008 17:35:13 -0000 1.64
    public Listview ... {
      ...
      private Image imgView = null;
      ...
      public void handleResize(boolean bForce){
        ...
        if(imgView == null || bForce){
          ...
+         if (imgView != null && !imgView.isDisposed())
+           imgView.dispose();
          imgView = new Image(...);
          ...


Testing object, test case

Testing object

Size of target program
Java Native C
Files LOC Files LOC
bug1 2705 340.6K N/A N/A

bug1-object.tar.gz


Test case

Total Used Pass Fail
# of test cases 8 8 7 1
  • We created unit test cases of the ListView class of Azureus.
      • The failing test case is constructed based on the error scenario described in the QVM paper.

    bug1-testcases.csv

      • 1st column : Test case - Name of test case
      • 2nd column : Test result - 0(fail) or 1(pass)
      • 3rd column : Source file - Location of source file containing test code



    Mutant generation

    Mutant

    Total generated by
    existing mutation operators    		 generated by
    new mutation operators
    # of mutants 2861 2710 151


    Mutation operator

    Total Existing New
    # of mutation operators 14 13 1

    bug1-mutants.csv

      • 1st column : Mutant ID - Unique ID for mutant
      • 2nd column : Mutated Line - Mutated line of testing object
      • 3rd column : Mutation Operator - Mutation operator applied to the mutated line


    Experiment result

    Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)

    bug1-result-susp.csv

      • 1st column : Line - Line of testing object
      • 2nd column : MuseSusp - Suspiciousness score of MUSEUM
      • 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
      • 4th column : JaccardSusp - Suspiciousness score of Jaccard
      • 5th column : JaccardRank - Suspiciousness ranking of Jaccard
      • 6th column : OchiaiSusp - Suspiciousness score of Ochiai
      • 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
      • 8th column : Op2Susp - Suspiciousness score of Op2
      • 9th column : Op2Rank - Suspiciousness ranking of Op2


    Mutation testing result

    make a passing test case fails
    (breaking)    		 make a failing test case passes
    (partial fix)    		 golden mutants
    (complete fix)
    # of mutants 305 1 1

    bug1-result-mutants.csv

      • 1st column : Line - Mutated line of testing object
      • 2nd column : LineCovTests - # of test cases that covers the mutated line
      • 3rd column : Pass - # of passing test cases in LineCovTests
      • 4th column : Fail - # of failing test cases in LineCovTests
      • 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
      • 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
      • 7th column : Mutants - a list of generated mutants

    Target source code, bug report, bug fix

    Target source code

    https://bitbucket.org/xerial/sqlite-jdbc (commit 8e825f7, sqlite-jdbc-3.7.8.jar)


    Bug report

    https://bitbucket.org/xerial/sqlite-jdbc/issue/16


    Bug fix
    The bug is fixed by the developer at the following commit

    https://bitbucket.org/xerial/sqlite-jdbc/commits/1568679fe438/

    Diff between
    1568679
    and
    8e825f7
    === modified file 'src/main/java/org/sqlite/Stmt.java'
        public int executedUpdate(String sql){
          ...
    +     changes = db.total_changes();
          ...
          db._exec(sql);
    -     changes = db.changes();
    +     changes = db.total_changes() - changes;
          ...
          return changes;
        }


    Testing object, test case

    Testing object

    Size of target program
    Java Native C
    Files LOC Files LOC
    bug2 20 4.6K 3 1.8K

    bug2-object.zip


    Test case

    Total Used Pass Fail
    # of test cases 151 150 149 1
  • 1 test case is excluded because the test case contains empty body.
  • We created one failing test case based on the bug report and the existing test cases.
      • The failing test case has an assertion that the return value of executeUpdate() with a DDL query must be zero.
      • Although the bug report provides a new test case, we cannot use the test case because it does not properly check that executeUpdate() works correctly with a DDL query.

    bug2-testcases.csv

      • 1st column : Test case - Name of test case
      • 2nd column : Test result - 0(fail) or 1(pass)
      • 3rd column : Source file - Location of source file containing test code



    Mutant generation

    Mutant

    Total generated by
    existing mutation operators    		 generated by
    new mutation operators
    # of mutants 691 603 88


    Mutation operator

    Total Existing New
    # of mutation operators 49 37 12

    bug2-mutants.csv

      • 1st column : Mutant ID - Unique ID for mutant
      • 2nd column : Mutated Line - Mutated line of testing object
      • 3rd column : Mutation Operator - Mutation operator applied to the mutated line


    Experiment result

    Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)

    bug2-result-susp.csv

      • 1st column : Line - Line of testing object
      • 2nd column : MuseSusp - Suspiciousness score of MUSEUM
      • 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
      • 4th column : JaccardSusp - Suspiciousness score of Jaccard
      • 5th column : JaccardRank - Suspiciousness ranking of Jaccard
      • 6th column : OchiaiSusp - Suspiciousness score of Ochiai
      • 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
      • 8th column : Op2Susp - Suspiciousness score of Op2
      • 9th column : Op2Rank - Suspiciousness ranking of Op2


    Mutation testing result

    make a passing test case fails
    (breaking)    		 make a failing test case passes
    (partial fix)    		 golden mutants
    (complete fix)
    # of mutants 555 3 0

    bug2-result-mutants.csv

      • 1st column : Line - Mutated line of testing object
      • 2nd column : LineCovTests - # of test cases that covers the mutated line
      • 3rd column : Pass - # of passing test cases in LineCovTests
      • 4th column : Fail - # of failing test cases in LineCovTests
      • 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
      • 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
      • 7th column : Mutants - a list of generated mutants

    Target source code, bug report, bug fix

    Target source code

    https://bitbucket.org/xerial/sqlite-jdbc (commit 9185323, sqlite-jdbc-3.7.15.jar)


    Bug report

    https://bitbucket.org/xerial/sqlite-jdbc/issue/36


    Bug fix
    The bug is fixed by the developer at the following commit

    https://bitbucket.org/xerial/sqlite-jdbc/commits/9185323

    Diff between
    5a06a06
    and
    9185323
    === modified file 'src/main/java/org/sqlite/PrepStmt.java'
       final class PrepStmt {
         ...
         public void clearParameters() throws SQLException {
           checkOpen();
    -      db.reset(pointer);
    -      clearBatch();
    +      db.clear_bindings(pointer);
    +      batch = null;
         }
       }


    Testing object, test case

    Testing object

    Size of target program
    Java Native C
    Files LOC Files LOC
    bug3 19 4.2K 2 1.7K

    bug3-object.zip


    Test case

    Total Used Pass Fail
    # of test cases 159 159 158 1
  • 1 test case is excluded because the test case contains empty body.

    • bug3-testcases.csv

      • 1st column : Test case - Name of test case
      • 2nd column : Test result - 0(fail) or 1(pass)
      • 3rd column : Source file - Location of source file containing test code



    Mutant generation

    Mutant

    Total generated by
    existing mutation operators    		 generated by
    new mutation operators
    # of mutants 965 867 98


    Mutation operator

    Total Existing New
    # of mutation operators 53 41 12

    bug3-mutants.csv

      • 1st column : Mutant ID - Unique ID for mutant
      • 2nd column : Mutated Line - Mutated line of testing object
      • 3rd column : Mutation Operator - Mutation operator applied to the mutated line


    Experiment result

    Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)

    bug3-result-susp.csv

      • 1st column : Line - Line of testing object
      • 2nd column : MuseSusp - Suspiciousness score of MUSEUM
      • 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
      • 4th column : JaccardSusp - Suspiciousness score of Jaccard
      • 5th column : JaccardRank - Suspiciousness ranking of Jaccard
      • 6th column : OchiaiSusp - Suspiciousness score of Ochiai
      • 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
      • 8th column : Op2Susp - Suspiciousness score of Op2
      • 9th column : Op2Rank - Suspiciousness ranking of Op2


    Mutation testing result

    make a passing test case fails
    (breaking)    		 make a failing test case passes
    (partial fix)    		 golden mutants
    (complete fix)
    # of mutants 793 7 1

    bug3-result-mutants.csv

      • 1st column : Line - Mutated line of testing object
      • 2nd column : LineCovTests - # of test cases that covers the mutated line
      • 3rd column : Pass - # of passing test cases in LineCovTests
      • 4th column : Fail - # of failing test cases in LineCovTests
      • 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
      • 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
      • 7th column : Mutants - a list of generated mutants

    Target source code, bug report, bug fix

    Target source code

    http://ftp.gnome.org/pub/gnome/sources/java-gnome/4.0/java-gnome-4.0.10.tar.gz


    Bug report

    https://bugzilla.gnome.org/show_bug.cgi?id=576111


    Bug fix
    The reporter suggested a quick fix at the bug report.

    https://bugzilla.gnome.org/show_bug.cgi?id=576111

    >>> Fix <<<
    A quick fix is to make the "receiver" global reference.
    src/jni/bindings_java_signal.c:
      524 bjc->receiver = (*env)->NewGlobalRef(env,receiver);
    The fix has not been confirmed by developer.


    Testing object, test case

    Testing object

    Size of target program
    Java Native C
    Files LOC Files LOC
    bug4 1097 64.2K 496 65.6K

    bug4-object.zip


    Test case

    Total Used Pass Fail
    # of test cases 176 170 169 1
  • We used 1 failing test case from 7 failing test cases.
      • The one that used for the fist error-reproducing test case in the bug report
      • All 7 failing test cases are infected by the same bug (not the same errors) because the same fix resolve all errors

    bug4-testcases.csv

      • 1st column : Test case - Name of test case
      • 2nd column : Test result - 0(fail) or 1(pass)
      • 3rd column : Source file - Location of source file containing test code



    Mutant generation

    Mutant

    Total generated by
    existing mutation operators    		 generated by
    new mutation operators
    # of mutants 718 677 41


    Mutation operator

    Total Existing New
    # of mutation operators 47 37 10

    bug4-mutants.csv

      • 1st column : Mutant ID - Unique ID for mutant
      • 2nd column : Mutated Line - Mutated line of testing object
      • 3rd column : Mutation Operator - Mutation operator applied to the mutated line


    Experiment result

    Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)

    bug4-result-susp.csv

      • 1st column : Line - Line of testing object
      • 2nd column : MuseSusp - Suspiciousness score of MUSEUM
      • 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
      • 4th column : JaccardSusp - Suspiciousness score of Jaccard
      • 5th column : JaccardRank - Suspiciousness ranking of Jaccard
      • 6th column : OchiaiSusp - Suspiciousness score of Ochiai
      • 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
      • 8th column : Op2Susp - Suspiciousness score of Op2
      • 9th column : Op2Rank - Suspiciousness ranking of Op2


    Mutation testing result

    make a passing test case fails
    (breaking)    		 make a failing test case passes
    (partial fix)    		 golden mutants
    (complete fix)
    # of mutants 358 2 2

    bug4-result-mutants.csv

      • 1st column : Line - Mutated line of testing object
      • 2nd column : LineCovTests - # of test cases that covers the mutated line
      • 3rd column : Pass - # of passing test cases in LineCovTests
      • 4th column : Fail - # of failing test cases in LineCovTests
      • 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
      • 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
      • 7th column : Mutants - a list of generated mutants

    Target source code, bug report, bug fix

    Target source code

    http://research.operationaldynamics.com/bzr/java-gnome/mainline/ (revision 658)


    Bug report
    Commit log

    Contributor Andrew Cowie
    Date 07-June-2009 at 09:54
    Commit Comment Avoid segfault lurking in GtkSpell library

    It seems the implementation in the GtkSpell library has a signal
    handler connected to the GtkTextView 'destroy' signal calling its
    internal gtkspell_free() function. If we call gtkspell_detach() it
    results in a segmentation fault.


    Bug fix
    The bug is fixed by the developer at revision 659

    http://research.operationaldynamics.com/bzr/java-gnome/mainline/ (revision 659)

    Diff between
    rev. 658
    and
    rev. 659
    === modified file 'src/bindings/org/gnome/gtk/Spell.java'
    --- src/bindings/org/gnome/gtk/Spell.java 2009-06-07 11:00:30 +0000
    +++ src/bindings/org/gnome/gtk/Spell.java 2009-06-07 13:54:01 +0000
        protected void release() {
    -      GtkSpell.detach(this);
    +   // GtkSpell.detach(this);
        }


    Testing object, test case

    Testing object

    Size of target program
    Java Native C
    Files LOC Files LOC
    bug5 1134 67.1K 514 69.2K

    bug5-object.zip


    Test case

    Total Used Pass Fail
    # of test cases 194 184 183 1
  • 10 test cases are excluded because of the test cases are duplicated
  • We used 1 failing test cases and 183 passing test cases
      • 1 failing test case: we wrote a failing test case based on the bug report and the existing test cases since no error-reproducing test case is given.
      • 183 passing test case: we used the regression test cases in the Java-gnome repository (revision 659) for passing test case executions.

    bug5-testcases.csv

      • 1st column : Test case - Name of test case
      • 2nd column : Test result - 0(fail) or 1(pass)
      • 3rd column : Source file - Location of source file containing test code



    Mutant generation

    Mutant

    Total generated by
    existing mutation operators    		 generated by
    new mutation operators
    # of mutants 369 340 29


    Mutation operator

    Total Existing New
    # of mutation operators 45 35 10

    bug5-mutants.csv

      • 1st column : Mutant ID - Unique ID for mutant
      • 2nd column : Mutated Line - Mutated line of testing object
      • 3rd column : Mutation Operator - Mutation operator applied to the mutated line


    Experiment result

    Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)

    bug5-result-susp.csv

      • 1st column : Line - Line of testing object
      • 2nd column : MuseSusp - Suspiciousness score of MUSEUM
      • 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
      • 4th column : JaccardSusp - Suspiciousness score of Jaccard
      • 5th column : JaccardRank - Suspiciousness ranking of Jaccard
      • 6th column : OchiaiSusp - Suspiciousness score of Ochiai
      • 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
      • 8th column : Op2Susp - Suspiciousness score of Op2
      • 9th column : Op2Rank - Suspiciousness ranking of Op2


    Mutation testing result

    make a passing test case fails
    (breaking)    		 make a failing test case passes
    (partial fix)    		 golden mutants
    (complete fix)
    # of mutants 311 51 15

    bug5-result-mutants.csv

      • 1st column : Line - Mutated line of testing object
      • 2nd column : LineCovTests - # of test cases that covers the mutated line
      • 3rd column : Pass - # of passing test cases in LineCovTests
      • 4th column : Fail - # of failing test cases in LineCovTests
      • 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
      • 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
      • 7th column : Mutants - a list of generated mutants

    Target source code, bug report, bug fix

    Target source code
    Target version is SWT-3.7.0.3

    http://git.eclipse.org/c/platform/eclipse.platform.swt.git/snapshot/eclipse.platform.swt-7f81e1378bd9260f3e4479e4ddec2d1f30e778f3.tar.gz (eclipse swt 3.7.0.3)


    Bug report
    The bug is first reported at SWT-3.5.0 by the developers

    http://bugs.eclipse.org/bugs/show_bug.cgi?id=322222


    Bug fix
    The patch is applied for SWT-3.7.0.4 by the developers

    http://git.eclipse.org/c/platform/eclipse.platform.swt.git/commit/?id=5b6a9aaed8ec8374fe407677cec6e0cafb2d185d

    Diff between
    commit 5b6a9a
    and
    commit 7f81e1
    === modified file 'gtk/org/eclipse/swt/widgets/Display.java'
       public class Display {
         ...
         void initializeSubclasses() {
           if (OS.GTK_VERSION >= OS.VERSION (2, 4, 0)) {
             OS.G_OBJECT_CLASS_CONSTRUCTOR(PLClass);
             OS.G_OBJECT_CLASS_SET_CONSTRUCTOR(PLClass, NewProc);
             p = OS.G_OBJECT_CLASS_CONSTRUCTOR(PLClass);
             OS.G_OBJECT_CLASS_SET_CONSTRUCTORR(PLClass, new NewProcCB(p));
         ...


    Testing object, test case

    Testing object

    Size of target program
    Java Native C
    Files LOC Files LOC
    bug6 582 118.7K 29 20.7K

    bug6-object.zip


    Test case

    Total Used Pass Fail
    # of test cases 50 50 49 1
    • We used total 49 test cases with 1 failing test case
      • 1 failing test case: the test case was given with a bug report, and used for the Blink paper.
        • 48 passing test cases: we chose the test cases for the Display module only because all error-stack traces in the duplicated bug reports commonly have at least one Display method.

    bug6-testcases.csv

      • 1st column : Test case - Name of test case
      • 2nd column : Test result - 0(fail) or 1(pass)
      • 3rd column : Source file - Location of source file containing test code



    Mutant generation

    Mutant

    Total generated by
    existing mutation operators    		 generated by
    new mutation operators
    # of mutants 9479 8926 553


    Mutation operator

    Total Existing New
    # of mutation operators 79 66 13

    bug6-mutants.csv

      • 1st column : Mutant ID - Unique ID for mutant
      • 2nd column : Mutated Line - Mutated line of testing object
      • 3rd column : Mutation Operator - Mutation operator applied to the mutated line


    Experiment result

    Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)

    bug6-result-susp.csv

      • 1st column : Line - Line of testing object
      • 2nd column : MuseSusp - Suspiciousness score of MUSEUM
      • 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
      • 4th column : JaccardSusp - Suspiciousness score of Jaccard
      • 5th column : JaccardRank - Suspiciousness ranking of Jaccard
      • 6th column : OchiaiSusp - Suspiciousness score of Ochiai
      • 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
      • 8th column : Op2Susp - Suspiciousness score of Op2
      • 9th column : Op2Rank - Suspiciousness ranking of Op2


    Mutation testing result

    make a passing test case fails
    (breaking)    		 make a failing test case passes
    (partial fix)    		 golden mutants
    (complete fix)
    # of mutants 3612 17 14

    bug6-result-mutants.csv

      • 1st column : Line - Mutated line of testing object
      • 2nd column : LineCovTests - # of test cases that covers the mutated line
      • 3rd column : Pass - # of passing test cases in LineCovTests
      • 4th column : Fail - # of failing test cases in LineCovTests
      • 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
      • 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
      • 7th column : Mutants - a list of generated mutants

    Target source code, bug report, bug fix

    Target source code

    https://bitbucket.org/xerial/sqlite-jdbc (sqlite-jdbc-3.6.0.jar)


    Bug report

    Lee et. al. "Debugging Mixed-Environment Programs with Blink"


    Bug fix
    The bug is fixed by the developer at the following commit

    https://github.com/xerial/sqlite-jdbc/commit/847f3645401afb0d80a8a18340eb5e5cd5abb32d

    Diff between
    847f364
    and
    c4e0e7c
    === modified file 'src/main/java/org/sqlite/NativeDB.c'
       static xCall (...) {
         ...
    +     // check if xFunc threw an Exception
    +     if (*env)->ExceptionCheck(env)) {
    +       xFunc_error(context, env);
    +     }
         (*env)->SetLongField(env, func, fld_context, 0);
         (*env)->SetLongField(env, func, fld_value, 0);
         (*env)->SetIntField(env, func, fld_args, 0);
    -     // check if xFunc threw an Exception
    -     if ((*env)->ExceptionCheck(env)) xFunc_error(context, env);


    Testing object, test case

    Testing object

    Size of target program
    Java Native C
    Files LOC Files LOC
    bug3 25 4.9K 2 0.6K

    bug7-object.zip


    Test case

    Total Used Pass Fail
    # of test cases 112 112 111 1

    bug7-testcases.csv

      • 1st column : Test case - Name of test case
      • 2nd column : Test result - 0(fail) or 1(pass)
      • 3rd column : Source file - Location of source file containing test code



    Mutant generation

    Mutant

    Total generated by
    existing mutation operators    		 generated by
    new mutation operators
    # of mutants 844 700 144


    Mutation operator

    Total Existing New
    # of mutation operators 52 40 12

    bug7-mutants.csv

      • 1st column : Mutant ID - Unique ID for mutant
      • 2nd column : Mutated Line - Mutated line of testing object
      • 3rd column : Mutation Operator - Mutation operator applied to the mutated line


    Experiment result

    Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)

    bug7-result-susp.csv

      • 1st column : Line - Line of testing object
      • 2nd column : MuseSusp - Suspiciousness score of MUSEUM
      • 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
      • 4th column : JaccardSusp - Suspiciousness score of Jaccard
      • 5th column : JaccardRank - Suspiciousness ranking of Jaccard
      • 6th column : OchiaiSusp - Suspiciousness score of Ochiai
      • 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
      • 8th column : Op2Susp - Suspiciousness score of Op2
      • 9th column : Op2Rank - Suspiciousness ranking of Op2


    Mutation testing result

    make a passing test case fails
    (breaking)    		 make a failing test case passes
    (partial fix)    		 golden mutants
    (complete fix)
    # of mutants 542 3 3

    bug7-result-mutants.csv

      • 1st column : Line - Mutated line of testing object
      • 2nd column : LineCovTests - # of test cases that covers the mutated line
      • 3rd column : Pass - # of passing test cases in LineCovTests
      • 4th column : Fail - # of failing test cases in LineCovTests
      • 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
      • 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
      • 7th column : Mutants - a list of generated mutants

    Target source code, bug report, bug fix

    Target source code
    Target version is SWT-4.4

    http://git.eclipse.org/c/platform/eclipse.platform.swt.git/ (Eclipse SWT 4.4)


    Bug report
    The bug is first reported by the developers and has not resolved (open bug)

    https://bugs.eclipse.org/bugs/show_bug.cgi?id=419729


    Testing object, test case

    Testing object

    Size of target program
    Java Native C
    Files LOC Files LOC
    bug8 591 126.6K 29 48.5K

    bug8-object.zip


    Test case

    Total Used Pass Fail
    # of test cases 204 204 203 1

      bug8-testcases.csv

        • 1st column : Test case - Name of test case
        • 2nd column : Test result - 0(fail) or 1(pass)
        • 3rd column : Source file - Location of source file containing test code



      Mutant generation

      Mutant

      Total generated by
      existing mutation operators      		 generated by
      new mutation operators
      # of mutants 14490 13450 1040


      Mutation operator

      Total Existing New
      # of mutation operators 59 43 16

      bug8-mutants.csv

        • 1st column : Mutant ID - Unique ID for mutant
        • 2nd column : Mutated Line - Mutated line of testing object
        • 3rd column : Mutation Operator - Mutation operator applied to the mutated line


      Experiment result

      Suspiciousness score, ranking (MUSEUM, Jaccard, Ochiai, Op2)

      bug8-result-susp.csv

        • 1st column : Line - Line of testing object
        • 2nd column : MuseSusp - Suspiciousness score of MUSEUM
        • 3rd column : MuseRank - Suspiciousness ranking of MUSEUM
        • 4th column : JaccardSusp - Suspiciousness score of Jaccard
        • 5th column : JaccardRank - Suspiciousness ranking of Jaccard
        • 6th column : OchiaiSusp - Suspiciousness score of Ochiai
        • 7th column : OchiaiRank - Suspiciousness ranking of Ochiai
        • 8th column : Op2Susp - Suspiciousness score of Op2
        • 9th column : Op2Rank - Suspiciousness ranking of Op2


      Mutation testing result

      make a passing test case fails
      (breaking)      		 make a failing test case passes
      (partial fix)      		 golden mutants
      (complete fix)
      # of mutants 8766 1 1

      bug8-result-mutants.csv

        • 1st column : Line - Mutated line of testing object
        • 2nd column : LineCovTests - # of test cases that covers the mutated line
        • 3rd column : Pass - # of passing test cases in LineCovTests
        • 4th column : Fail - # of failing test cases in LineCovTests
        • 5th column : Fail2Pass - # of test cases in LineCovTests that fails on original program but passes on the mutant
        • 6th column : Pass2Fail - # of test cases in LineCovTests that passes on original program but fails on the mutant
        • 7th column : Mutants - a list of generated mutants