1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
|
DexFuzz
=======
DexFuzz is primarily a tool for fuzzing DEX files. Fuzzing is the introduction of
subtle changes ("mutations") to a file to produce a new test case. These test cases
can be used to test the various modes of execution available to ART (Interpreter,
Quick compiler, Optimizing compiler) to check for bugs in these modes of execution.
This is done by differential testing - each test file is executed with each mode of
execution, and any differences between the resulting outputs may be an indication of
a bug in one of the modes.
For a wider overview of DexFuzz, see:
http://community.arm.com/groups/android-community/blog/2014/11/26/the-art-of-fuzz-testing
In typical operation, you provide DexFuzz with a set of DEX files that are the "seeds"
for mutation - e.g. some tests taken from the ART test suite - and point it at an
ADB-connected Android device, and it will fuzz these seed files, and execute the
resulting new tests on the Android device.
How to run DexFuzz
==================
1. Build dexfuzz with mmm tools/dexfuzz from within art/.
2. Make sure you have an Android device connected via ADB, that is capable of
having DEX files pushed to it and executed with the dalvikvm command.
3. Make sure you're in the Android build environment!
(That is, . build/envsetup.sh && lunch)
4. Create a new directory, and place some DEX files in here. These are the seed files
that are mutated to form new tests.
5. Create a directory on your device that mutated test files can be pushed to and
executed in, using dalvikvm. For example, /data/art-test/
6. If you currently have multiple devices connected via ADB, find out the name of
your device using "adb devices -l".
7. Run this command:
dexfuzz --inputs=<seeds dir> --execute --repeat=<attempts> \
--dump-output <combination of ISA(s) and and backend(s)>
You MUST specify one of the following ISAs:
--arm
--arm64
--x86
--x86_64
--mips
--mips64
And also at least two of the following backends:
--interpreter
--quick
--optimizing
Note that if you wanted to test both ARM and ARM64 on an ARM64 device, you can use
--allarm. Also in this case only one backend is needed, if i.e., you wanted to test
ARM Quick Backend vs. ARM64 Quick Backend.
Some legal examples:
--arm --quick --optimizing
--x86 --quick --optimizing --interpreter
--allarm --quick
Add in --device=<device name, e.g. device:generic> if you want to specify a device.
Add in --execute-dir=<dir on device> if you want to specify an execution directory.
(The default is /data/art-test/)
As the fuzzer works, you'll see output like:
|-----------------------------------------------------------------|
|Iterations|VerifyFail|MutateFail|Timed Out |Successful|Divergence|
|-----------------------------------------------------------------|
| 48 | 37 | 4 | 0 | 6 | 1 |
Iterations - number of attempts we've made to mutate DEX files.
VerifyFail - the number of mutated files that ended up failing to verify, either
on the host, or the target.
MutateFail - because mutation is a random process, and has attempt thresholds to
avoid attempting to mutate a file indefinitely, it is possible that
an attempt to mutate a file doesn't actually mutate it. This counts
those occurrences.
Timed Out - mutated files that timed out for one or more backends.
Current timeouts are:
Quick - 5 seconds
Optimizing - 5 seconds
Intepreter - 30 seconds
(use --short-timeouts to set all backends to 2 seconds.)
Successful - mutated files that executed and all backends agreed on the resulting
output. NB: if all backends crashed with the same output, this would
be considered a success - proper detection of crashes is still to come.
Divergence - mutated files that executed and some backend disagreed about the
resulting output. Divergent programs are run multiple times with a
single backend, to check if they diverge from themselves, and these are
not included in the count. If multiple architectures are being used
(ARM/ARM64), and the divergences align with different architectures,
these are also not included in the count.
8. Check report.log for the full report, including input file and RNG seed for each
test program. This allows you to recreate a bad program with, e.g.:
dexfuzz --input=<input file> --seed=<seed value>
Check dexfuzz --help for the full list of options.
NOTE: DEX files with unicode strings are not fully supported yet, and DEX files with
JNI elements are not supported at all currently.
Mutation Likelihoods
====================
Each bytecode mutation has a chance out of 100% of firing. Following is the listing
of each mutation's probability. If you wish to easily adjust these values, copy
these values into a file called likelihoods.txt, and run dexfuzz with
--likelihoods=likelihoods.txt.
ArithOpChanger 75
BranchShifter 30
CmpBiasChanger 30
ConstantValueChanger 70
ConversionRepeater 50
FieldFlagChanger 40
InstructionDeleter 40
InstructionDuplicator 80
InstructionSwapper 80
NewMethodCaller 10
NonsenseStringPrinter 10
PoolIndexChanger 30
RandomInstructionGenerator 30
SwitchBranchShifter 30
TryBlockShifter 40
ValuePrinter 40
VRegChanger 60
|
