
=======================================================
PC Week 1999 Database Server Benchmark, ver. 2.7 Readme
=======================================================


Table of Contents
=================

1. Use of these files
2. Acknowledgements
3. What you need to run the benchmark
4. The files that make up this benchmark
5. Configuring your server
6. Configuring your clients
7. Installing the benchmark scripts
8. Configurating the benchmark scripts
9. Generating the data
10. Running the benchmark
11. Fixes and further questions


=====================
1. Use of these files
=====================

This is PC Week Labs' SQL server database benchmark kit. We've developed this package to help us measure database performance under controlled, repeatable conditions. It includes a wide variety of tests to measure how a database performs under a wide variety of common database activities.

We are providing this code in order to provide a full disclosure of how we measure database performance. Our readers can also use this kit to do their own internal benchmarking, and vendors are also free to use this kit for performance tuning in preparation for a PC Week benchmark.

However, publishing results from these benchmark files is not allowed. If you want to do so, please contact PC Week Labs (contact information is at the end of this file) and we may be able to arrange a formal test.

The reason why we need this restriction is that proper comparative benchmarking is a combination of benchmark design and carefully controlled benchmark execution. As such, these files are only part of what is needed for a valid cross-product performance comparison. Since this test has our name on it, we will only allow publication of results from it when we personally set up the testbed and run the tests.


===================
2. Acknowledgements
===================

Since it is based on the AS3AP benchmark, this benchmark has nearly a twenty year history of study and use in the database industry.

Dina Bitton, Jeff Millman, Cyril Orji and Carolyn Turbyfill at Cornell University and at the University of Illinois at Chicago developed the original AS3AP benchmark upon which this code is based, as well as the AS3AP data generator we are including with in this benchmark (as3apgen.exe).

We thank Dr. Bitton (now at IDS Integrated Data Systems Corp, www.ids-corp.com) for allowing us to distribute the data generator with our benchmark. The data generator is a Win32 executable.

Brian Butler (now of Client/Server Solutions) revamped the original AS3AP tests in 1994, adding much better write testing and an expanded query set.


=====================================
3. What you need to run the benchmark
=====================================

a) You need a database server with the following disk space requirements.

If you want to run the full 4.4 GB dataset test, you will need:

- At least 4.5 GB free on drive D or /dbimportdata (for the generated import data files)
- At least 1.0 GB free on drive E or /dblog (for the database transaction log)
- At least 10.3 GB free on drive F or /dbdata (for the database data devices)

If you want to run the full 44 KB dataset test, you will need:

- At least 44 MB free on drive D or /dbimportdata (for the generated import data files)
- At least 10 MB free on drive E or /dblog (for the database transaction log)
- At least 103 MB free on drive F or /dbdata (for the database data devices)

You can place files in different locations than those above, but will then need to edit the script files to point to the right places. Instructions on doing this are in the "Installing the benchmark" section below.

b) You need an installed copy of one of the supported databases:

- InterBase 5.6 (code is in Code/InterBase)
- Microsoft SQL Server 7.0 (code is in Code/MSSQLBench)
- Oracle8i 8.1.5 (code is in Code/OracleBench)
- PostgreSQL 6.5.3 (code is in Code/PostgreSQL)

c) You need a copy of Client/Server Solutions Inc.'s Benchmark Factory 2.0 (www.benchmarkfactory.com) with the appropriate driver (ODBC or selected native driver).


========================================
4. The files that make up this benchmark
========================================

- The "Docs" directory

Documentation for the benchmark is in the "Documentation" directory. Within this directory, there are a set of documents that provide a high-level description of the benchmark (the files in the "Part I - Description" directory) and a low-level description, including table, index and SQL query information (the files in the "Part II - Implementation" directory).

Documentation files are provided in Microsoft Word 97/2000, Microsoft Excel 97/2000 and HTML formats.

- The "Code" directory

Database scripts to set up the benchmark databases are in the "Code" directory.

These database scripts are grouped into nine steps (the nine steps of running the benchmark). They are:

a) Generating the benchmark data
b) Configuring the database server for optimal performance
c) Creating the benchmark database
d) Loading the benchmark data (from step A) into the database
e) Indexing the database
f) Checking the database for correctness and creating integrity-check tables
g) Running the transaction mixes
h) Checking the database again to ensure it is still correct
i) Deleting the database (to clean up)

The Windows NT batch files/bash scripts and SQL scripts that perform these operations are grouped into directories corresponding to the steps above.

To run the benchmark, we've included compiled Windows NT .DLL versions of this benchmark (in the "BFactory DLLs" directory). Source code and a Microsoft Visual C++ 6.0 project file for these DLL files are provided so you can examine or modify the code.

IMPORTANT NOTE: These implementations are sample implementations we developed internally. They have not been optimized (or even seen) by Microsoft or Oracle engineers, and may not provide the fastest possible performance (but they are intended to be as fast as possible).


==========================
5. Configuring your server
==========================

Our scripts are optimized for a particular benchmark configuration, specifically a 2 CPU server with 512MB of RAM and four disk volumes, and a TCP/IP-based network.

If you have a different hardware or drive configuration from what we used, you will need to modify the scripts as described in the "Configuring the benchmark" section below to match your hardware setup.

The optimal way to arrange hard drives for this test is as follows:

Drive C or / (a single disk) stores the:
- operating system
- database software
- benchmark scripts

Drive C or swap partition (the same disk as above) stores the:
- operating system swap file (at least 2 X RAM size)

Drive D or /dbimportdata (a single disk or RAID volume) stores the:
- generated import data files

Drive E or /dblog (a single disk or RAID volume) stores the:
- database transaction log

Drive F or /dbdata (a single RAID volume of as many disks as possible) stores the:
- database data devices

The exact hardware and software configuration settings we used are in the part I of the benchmark documentation.

For InterBase, when installing the database:

- Perform an all-defaults installation logged in as the user "interbase" belonging to group "interbase". Edit .bash_profile to add "/usr/local/interbase/bin" to your PATH environment variable. Also set the environment variables to the following values:

"export INTERBASE=/usr/local/interbase"

For Microsoft SQL Server, when installing the database:

- Perform an all-defaults installation except use the "Binary" sort order (this is the fastest option, and Oracle's default sort order).

For Oracle, when installing the database:

- Perform an all-defaults installation (if you don't install to "c:\Oracle" you will need to modify scripts as described in the "Configurating the benchmark" section below).

For PostgreSQL, when installing the database:

- Perform an all-defaults installation logged in as the user "postgres" belonging to group "postgres". Edit .bash_profile to add "/usr/local/pgsql/bin" to your PATH environment variable and add "/usr/local/pgsql/man" to your MANPATH environment variable. Also set the following environment variables to the following values:

"export PGLIB=/usr/local/pgsql/lib"
"export PGDATA=/dbdata/pgsql"

You'll also need to install the Benchmark Factory Visual Control Center on your benchmark controller server (which should not be the same machine as the database server to avoid resource contention) and create a Benchmark Factory profile pointing to the database you are testing. There are other instructions on how to configure the Visual Control Center listed in the "Benchmark Factory Settings" section of part II of the benchmark documentation (the implementation document).


===========================
6. Configuring your clients
===========================

For InterBase, when installing the clients:

- Use the ODBC driver. Set the "Database Name" field to (ip_addr_of_db_server):/dbdata/interbase/pcweek.gdb. In the "Advanced" tab, set "Lock Time Out" to "-1 - Wait" and check the "Application Using Threads" option. Leave other options at default values.

For Microsoft SQL Server, when installing the clients:

- Set the default client library on the clients to "TCP/IP" and add a TCP/IP server alias with the hostname of your database server (use the "Client Network Utility" program to do this)

(The client install for Microsoft SQL Server is hard to find. Choose "Install SQL Server 7.0 Components", "Database Server - Desktop Edition", then do a Custom install and only select the "Client Connectivity" option.)

For Oracle, when installing the clients:

- Do an all-defaults client installation and then add a Net Service Name called "pcmag" using TCP/IP with the hostname of your database server and a service name of "pcmag.pantheon.org" (use the "Net8 Easy Config" program to do this or do it as part of the client installation)

For PostgreSQL, when installing the clients:

- Use the ODBC driver. Set the "Database" field to "pcweek" and enter the server's IP address in the "Server" field. The "User Name" and "Password" fields should be set to "pcweek" and "pcweek", respectively. In the "Driver" (Advanced Options) page, check the "Recognize Unique Indexes" and "Use Declare/Fetch" options and uncheck the other six options in the top of the window. Set "Cache Size" to 100. Leave other options at default values.

Once you've installed the database client software, you'll also need to install Benchmark Factory's client on each of your client systems.


===================================
7. Installing the benchmark scripts
===================================

Unzip the contents of the "pcwdb27.zip" file to a temporary directory (IMPORTANT: use PKZIP's -d option or extract using WinZip's Action, Extract, Use folder names option to preserve the subdirectory structure in the zip file).

Then:

a) Copy the folder "A - Generate Data" to C:\

b) Copy the folder for the database you are testing to C:\ or ~.

If you are testing InterBase, untar InterBaseBench.tar.gz into the home directory of the user "interbase".

If you are testing Microsoft SQL Server, copy the folder "MSSQLBench" to C:\.

If you are testing Oracle, copy the folder "OracleBench" to c:\.

If you are testing PostgreSQL, untar PostgreSQLBench.tar.gz into the home directory of the user "postgres".

(If you put these files in different disk locations, you'll need to edit the script files they include so they know where they're located.)

c) Copy the .DLL file you want to use for testing from the folder "Benchmark Factory DLLs" to the Benchmark Factory "bin" directory (on both the Visual Control Center machine and on all the client systems) and register this DLL using Visual Control Center's Project, Settings, Benchmarks, Benchmark, Register Benchmark command (it's the small icon near the right side of the window).

By default, Benchmark Factory's bin directory is located at "C:\Program Files\CSS\Benchmark Factory\bin".


======================================
8. Configurating the benchmark scripts
======================================

There are a number of necessary modifications to get the benchmark scripts running on a system different than the one we used for our testing.

In particular, if you are placing data import files or database log or data files onto drive letters different than the ones specified in the "Configuring your server" section above, you'll need to edit the scripts to reflect those changes.

a) all platforms -- "A - Generate Data" directory:

- Edit "dbdatagen.bat" to set the PCM_IMPORT_DATA_DRIVE and PCM_IMPORT_DATA_DIRECTORY variables to the drive letter and directory where the data import files are located

b) InterBase and PostgreSQL:

- Edit "~/subroutines" to change where the script looks for its files.

c) Microsoft SQL Server and Oracle:

These are older scripts and require more work to customize.

Benchmark scripts home directory (normally "c:\MSSQLBench" or "c:\OracleBench"):

- For Microsoft SQL Server, edit "dbcheck_writes_mssql.bat", "dbundo_writes_mssql.bat", "runosql_pcmag.bat" and "runosql_sa.bat" to change the server TCP/IP hostname parameter from "-S mars" to "-S <your_server_hostname>"
- For Microsoft SQL Server, edit "runosql_sa.bat" to change the password parameter from "-P" to "-P <your_sa_password>" (the script is configured to submit a blank password for sa)
- For Oracle, edit "dbsetup_orcl.bat" to set the ORACLE_LOG_DRIVE and ORACLE_DATA_DRIVE variables to the correct drive letters (colons are required after the drive letters)
- For Oracle, if you did not install Oracle into "c:\Oracle", edit "dbsetup_orcl.bat" to change the references to "c:\Oracle" to where you have Oracle installed

"B - Configure Database Server" directory:

- For Microsoft SQL Server, edit "b2-tempdb_setup_70_a.sql" to change the reference to "f:\" to where you want database data devices to go, and change the reference to "e:\" to where you want the database log devices to go
- For Oracle, if you did not install Oracle into "c:\Oracle", edit "init.ora" and "initpcmag.ora" to change the references to "c:\Oracle" to where you have Oracle installed

"C - Create Benchmark Database" directory:

- For Microsoft SQL Server, edit "c1-create_pcmag_db.sql" to change the reference to "f:\" to where you want database data devices to go, and change the reference to "e:\" to where you want the database log devices to go
- For Oracle, edit "c1-create_pcmag_db1.sql" and "c1-create_pcmag_db2.sql" to change the references to "f:\" to where you want database data devices to go, and change the references to "e:\" to where you want the database log devices to go
- For Oracle, if you did not install Oracle into "c:\Oracle", edit "c1-create_pcmag_db1.sql" and "c1-create_pcmag_db2.sql" to change the references to "c:\Oracle" to where you have Oracle installed

"D - Load Database" directory:

- For Microsoft SQL Server, edit "d1-load_data.sql" to change the references to "d:\data\" to where you have placed the data import files
- For Oracle, edit "d1-load_data.bat" to change the references to "d:\data\" to where you have placed the data import files

"I - Delete Objects" directory:

- For Microsoft SQL Server, edit "reload_updates.bat" to change the server TCP/IP hostname parameter from "-S mars" to "-S <your_server_hostname>"
- For Microsoft SQL Server, edit "reload_updates.sql" to change the reference to "d:\data\" to where you have placed the data import files
- For Oracle, edit "reload_updates.bat" to change the reference to "d:\data\" to where you have placed the data import files
- For Oracle, edit "delete_entire_pcmag_database_and_instance.bat" to change the references to "f:\" to where you have database data devices  stored and change the references to "e:\" to where you have database log devices stored
- For Oracle, if you did not install Oracle into "c:\Oracle", edit "delete_entire_pcmag_database_and_instance.bat" to change the references to "c:\Oracle" to where you have Oracle installed


======================
9. Generating the data
======================

We aren't distributing the exact data set we used for our testing due to its size (about 4.5 GB). However, the data set can be regenerated as needed using the dbdatagen.bat batch file.

To generate the data, open a command prompt, change your current directory to "c:\A - Generate Data" and type:

	dbdatagen 44MB

or

	dbdatagen 4.4GB

The 4.43 GB version and is the version normally used for benchmarking. The small version of the data set is 44.4 MB of data and is used for desktop or workgroup databases or to quickly test that the database is correctly set up for benchmarking. (By default, the script places generated data on drive D.)

Note that since a newly generated data set will be different than the one we used, the following changes are needed:

- the expected number of rows and contents of those rows for each query listed in the "Query Analysis.xls" worksheet will be wrong for some queries
- output text files to the query mixes (the files end in "-MASTER.txt") used to check query correctness will need to be regenerated
- the stored procedure oltp_write_03_check need to be edited to reflect the correct answer to its check queries


=========================
10. Running the benchmark
=========================

Please see the Readme files specific to each database.


===============================
11. Fixes and further questions
===============================

I welcome any improvements to this suite should you care to send them to me.

Also, please feel free to send me an e-mail if you need help beyond what's in this file and in the benchmark documentation files.

Timothy Dyck
timothy_dyck@zd.com
Senior Analyst
PC Week Labs
