hive. Yahoo! is not affiliated with the authors of this page or responsible for its content.
Microsoft: Windows XP Reliability Study




October 2001
1001 Aviation Parkway, Suite 400 Morrisville, NC 27560 919-380-2800 Fax 919-380-2899
320 B Lakeside Drive Foster City, CA 94404 650-513-8000 Fax 650-513-8099
www.etestinglabs.com etesting_labs_info@ziffdavis.com 877-619-9259 (toll free)
Microsoft: Windows XP Reliability Study
Test report prepared under contract from Microsoft Corporation
Executive summary

Microsoft Corporation commissioned
eTesting Labs to measure the
reliability of Windows XP
Professional, Windows 2000
Professional Gold, and Windows 98
Second Edition Gold. We measured
the reliability of these three operating
systems by running application user
scripts on them continuously over
seven days to simulate a month of
typical home PC use. We counted
the number of application and
operating systems failures and found
that Windows XP Professional ran
over 30 times as long without
encountering problems as those
running Windows 98 SE.

We built a test bed on a private network consisting of twelve client systems and one primary domain controller
server. On four of the client systems we installed Windows 98 Second Edition Gold, on four other client
systems we installed Windows 2000 Professional Gold, and on the remaining four client systems we installed
Windows XP Professional, using the latest available updates. We installed Microsoft Office 2000 on all twelve
client systems and installed a user script to automate the applications that we installed on the client systems.
We configured the clients and server in the test bed to execute the user scripts, which they did for seven
consecutive days. For systems that had application or operating system failures, we attempted to fix them
using whatever means possible. Only as a last resort did we fix the failure by restarting the client systems. We
took notes that contained the specifics on how systems failed and what we did to get them executing the test
again.

Our results showed Windows XP Professional to be significantly more reliable than Windows 98 Second
Edition Gold. Although testing with different applications might show different results, our testing proves that
Windows XP Professional is significantly more reliable than Windows 98 SE. Windows XP Professional
performed without a single application or operating system failure during the entire test period whereas we
had to perform several system restarts on the Windows 98 SE systems.

After running the user scripts continuously over 7 days to simulate 31 days of typical home PC use, we found
the mean time to failure for Windows 98 Second Edition to be less than 4% of that of either Windows 2000
Key findings

‰Ã
Although testing with different applications might show
different results, in our testing systems running Windows XP
Professional ran over 30 times as long without encountering
problems as those running Windows 98 SE.
‰Ã
None of the Windows XP Professional systems had a single
application or operating system failure during the test period.
‰Ã
None of the Windows 2000 Professional Gold systems had a
single application or operating system failure during the test
period.


eTesting Labs: Microsoft Windows XP Reliability Study
2


Professional Gold or Windows XP Professional. We found the average of the maximum runtime to failure for
Windows 98 Second Edition Gold to be 15.7 equivalent user days, or roughly half that of the other two
operating systems. (Note that, due to system failures, Windows 98 Second Edition Gold systems may have
completed as few as 27 equivalent days of testing.)

Test results

Microsoft Corporation commissioned eTesting Labs to measure the reliability of Windows XP Professional,
Windows 2000 Professional Gold, and Windows 98 Second Edition Gold. We measured the reliability of these
three operating systems by running application user scripts on them continuously over seven days to simulate
a month of typical home PC use, and counting the number of times we had to reset the scripts manually or
had to restart the systems.

We created a test network containing twelve client systems and one server as described in the test
methodology section of this report. On four of the client systems we installed Windows 98 Second Edition
Gold, on four other client systems we installed Windows 2000 Professional Gold, and on the remaining four
client systems we installed Windows XP Professional, using the latest available updates. We installed
Microsoft Office 2000 on all twelve client systems and installed a user script to automate the applications that
we installed on the client systems. We configured the clients and server in the test bed to execute the user
scripts, which they did for seven consecutive days. For systems that had application or operating system
failures, we attempted to fix them using whatever means possible. Only as a last resort did we fix the failure
by restarting the client systems. We took notes that contained the specifics on how systems failed and what
we did to get them executing the test again.

After evaluating the user scripts, we determined that they generated 2,073 operations per iteration. See
Appendix C for the complete list of the 2,073 operations that the user scripts performed. Each user script
iteration took roughly one hour, regardless of the specific system performance. We kept detailed records on
the status of the test clients and summed the total number of operations that each client executed before
experiencing an application or operating system failure.

As described in the test methodology section, we ran the user scripts continuously over seven days to
simulate a month of typical home PC use. We used a value of 4.9 hours to represent a typical user day for
home PC usage, based on an internal study that Microsoft conducted. Using this equivalent user day, we
were able to simulate 31 equivalent user days of home PC usage in 7 days of testing on the Windows XP
Professional and the Windows 2000 Professional Gold systems. The Windows 98 Second Edition Gold
logged fewer iterations of the user scripts due to system failures, but all of them were able to simulate at least
27 equivalent user days of home PC usage in the 7-day test.

We monitored the status of the test systems at least twice a day during the week, typically once in the
morning and once in the evening, and once during the day on weekends. We did not provide 24-hour
surveillance on the test systems, so we did not see the failures as they occurred. Some failures may have
corrected themselves without detection and some failures may have worsened over time. Even though all of
the systems had similar start and end dates, system downtime due to failure prevented many of the systems
from executing comparable numbers of operations.

As shown in Figure 1, we were able to perform all 315,343 operations on all four of the Windows XP
Professional systems and on all four of the Windows 2000 Professional Gold systems.

As described above, a system that performed 315,343 operations performed 31.04 equivalent user days of
work. We calculated this as follows:

315,343 operations / 2073 operations per iteration * 1 iteration per hour / 4.9 hours per equivalent
user day = 31.04 equivalent user days

eTesting Labs: Microsoft Windows XP Reliability Study
3



All of these systems experienced no failures caused by either an application fault or operating system fault
during the entire test run. Each of the four Windows 98 Second Edition Gold systems experienced at least two
application failures and at least one operating system failure.

Test client
Total
Operations
User Days
Runtime
Failures
caused by
App
Failures
caused by
OS
Longest runtime
to
App failure
Longest runtime
to
OS failure
Win XP #1
315343 31.04
0
0
31.04
31.04
Win XP #2
315343 31.04
0
0
31.04
31.04
Win XP #3
315343 31.04
0
0
31.04
31.04
Win XP #4
315343 31.04
0
0
31.04
31.04
Win 2K #1
315343 31.04
0
0
31.04
31.04
Win 2K #2
315343 31.04
0
0
31.04
31.04
Win 2K #3
315343 31.04
0
0
31.04
31.04
Win 2K #4
315343 31.04
0
0
31.04
31.04
Win 98SE #1
294028 28.95
2
3
10.48
9.57
Win 98SE #2
288185 28.37
13
2
16.58
9.01
Win 98SE #3
319094 31.41
2
1
23.03
11.33
Win 98SE #4
275901 27.16
6
2
12.73
8.80
Figure 1: Test system failure statistics (Units are equivalent user days)

As shown in Figure 2, we found that the mean time to failure (MTTF) for Windows 98 Second Edition to be
significantly less than either Windows 2000 Professional Gold or Windows XP Professional.

Mean Time to Failure
124.2
124.2
124.2
124.2
124.2
124.2
12.9
4.8
3.6
0
20
40
60
80
100
120
140
MTTF (due to OS failure)
MTTF (due to App failure)
MTTF (Combined)
Equivalent User Days
Window s 98SE
Window s 2000
Window s XP

Figure 2: Mean time to failure across the three operating systems. Higher numbers reflect better
performance.


eTesti