Open Source Development Lab, Inc.
15275 SW Koll Parkway, Suite H
Beaverton, OR 97006

Phone: (503) 626-2455
Fax: (503) 626-2436
Email: info@osdl.org











Open Source Development Lab

Database Test 1









User Manual



June 18, 2002














Copyright (c) 2002 by The Open Source Development Laboratory, Inc. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0 or later (the latest version is currently available at http://www.opencontent.org/openpub/). Distribution of substantively modified versions of this document is prohibited without the explicit permission of the copyright holder.

Other company, product or service names may be trademarks or service marks of others.

Contributors to this white paper include:

Mark Wong (OSDL)

Jenny Zhang (OSDL)

Craig Thomas (OSDL)

Bryan Olmstead (OSDL)

Table of Contents

1Introduction 4

2Setup 4

2.1Getting and Installing the Files 4

2.1.1SAP DB 4

2.1.2OSDL-DBT-1 Test Kit Source 4

2.1.3Compiling the Test Kit 5

2.2Building the Database 5

2.2.1Sizing SAP DB 5

2.2.2Creating the Database 5

2.2.2.1SAP DB 5

3Configuration 6

3.1ODBC 6

4Running the Test Kit 6

5Test Results 7

6Advanced Topics 9

6.1Additional Compiler Options 9

6.2Manually Running the Test Kit 9

6.2.1Two-Tier Environment 10

6.2.2Three-Tier Environment 10

6.2.3Three-Tier Environment with Cache 11

6.3Tools 13

6.3.1ODBC Test Tools 13

6.3.2System Resources 15

6.3.3Database Notes 15

6.3.3.1SAP DB 15


1Introduction

This document provides instructions on how to set up and use the Open Source Development Lab's Database Transaction 1 (OSDL-DBT-1) test kit.

2Setup

This section covers the steps of getting and installing a database and the OSDL-DBT-1 test kit source code. The test kit currently only supports SAP DB.

2.1Getting and Installing the Files

2.1.1SAP DB

The OSDL-DBT-1 test kit was developed using SAP DB Version 7.3.0.21, however, newer versions of the database should work with the test kit. Linux binaries can be obtained from the Web at:

http://www.sapdb.org/tgz_linux.htm



The source package along with installation instructions can be found on the Web at:

http://www.sapdb.org/develop/dev_linux.htm



If you wish to install SAP DB with rpms, and if your Linux distribution supports rpm packages, they can be retrieved from the Web at:

http://www.sapdb.org/rpm_linux.htm

The following rpm packages are required:

Once the database has been installed, verify that the ODBC header files and libraries are included, as these are needed to build and install the OSDL-DBT-1 test kit. The rpm file sapdb-callif-7.3.0.21-1.i386.rpm needs to be installed on any system that communicates with the database as well as on the database system. Installing it onto the middle-tier system or the database system should be sufficient.

2.1.2OSDL-DBT-1 Test Kit Source

The latest stable version of the kit can be found on SourceForge via:

http://sourceforge.net/project/showfiles.php?group_id=52479



For complete instructions on retrieving the latest tested source from CVS see the Web at:

http://sourceforge.net/cvs/?group_id=52479

This is an abbrievated set of steps to download only the source code:

cvs -d:pserver:anonymous@cvs.osdldbt.sourceforge.net:/cvsroot/osdldbt login

cvs -z3 -d:pserver:anonymous@cvs.osdldbt.sourceforge.net:/cvsroot/osdldbt co dbt1

When prompted for a password, simply hit the enter key. Once the source has been downloaded and extracted, it is recommended to create a symbolic link from /usr/local/src/dbt1 to the location where the source code resides, since some scripts and programs assume the source code is in that location. For example:

ln -s /tmp/dbt1 /usr/local/dbt1

where /tmp/dbt1 is the location of the source code.

2.1.3Compiling the Test Kit

The test kit has different dependencies based on which database is used with the kit. For example, to use SAP DB, ODBC drivers must be also be installed in order to build the test kit. These details are explained in the previous sections for each database. Once those dependencies are met, the entire kit can be compiled by running make in the top level directory.

2.2Building the Database

The following sections explain how to build the database depending on which database is used. SAP DB is currently the only database supported.

2.2.1Sizing SAP DB

SAP DB can be used with any user account on the system.



An OpenOffice.org spreadsheet is provided to aid in sizing the database:

doc/sapdb_sizing.sxc



Under the Performance tab there are two parameters that can be changed. A default value of 1,000 is entered for the number of items to create and a default value of 1 is entered for the number of emulated users (EUs) expected to run against the database. The number of items can be changed to 10,000, 100,000, 1,000,000, or 10,000,000. The number of EUs can be any positive integer.



The data displayed under the Load tab report the expected size of the database, for the parameters entered, and are to be used for the physical database layout. The number to note is the total number of 8 KB pages since SAP DB uses those units in its configuration settings. Keep in mind that extra space must be taken into consideration because the database grows during a test run. The Load tab also calculates the growth rate of the database based on the expected number of users that access the database.

2.2.2Creating the Database

The scripts/ directory contains a list of databases that this test kit supports. Only SAP DB is currently supported. Under each of these database directories is a file called db_setup_sample.sh. This script has two variables defined in it, ITEMS and EUS, which refer to the number of items to generate in the database and the number of EUs expected to run. Executing the script runs the create_db.sh script, generates the database table data, creates the database tables, loads the database, creates indexes, loads the stored procedures, and backs up the database.

2.2.2.1SAP DB

The shell script, scripts/sapdb/create_db.sh, creates the physical database. In that script, there are lines beginning with param_adddevspace that define the devices on which the database resides. These lines must be tailored for the system devspace, log devspace, and user devspace in order to fit the I/O subsystem on the system where the database is running. The number defining the size of the log devspaces and the data devspaces are in units of 8 KB pages. This script is executed within the db_setup_sample.sh script that is explained in the next section.



The SAP DB database users dbm, dba, and dbt are created, with passwords the same as the user name, by create_db.sh. The user dbm is the database administrator account, dba is the utility account, and dbt is the user account. The dbt account is used when running the programs built in the test kit, or to execute SQL using the dbmcli program included with SAP DB.

3Configuration

This section explains how to configure various parts of the system before running the test kit.

3.1ODBC

Configuring ODBC may not be required depending on the database used with the test kit. With the current supported databases, ODBC is only used with SAP DB.



There are two ways to connect to the a database using ODBC. The first way is to use the address of the database system and the database name separated by a colon in for the server name. For example:

192.168.0.1:DBT1

is a valid server name for the DBT1 database on the system at 192.168.0.1. The hostname can also be used instead of an IP address.



The second method uses the .odbc.ini file that must reside in the home directory of the user attempting to run the program. The format of the file is:

[alias]

ServerDB = database_name

ServerNode = address

Driver = /opt/sapdb/interfaces/odbc/lib/libsqlod.so

Description = any text

For example:

[dbt1]

ServerDB = DBT1

ServerNode = 192.168.0.1

Driver = /opt/sapdb/interfaces/odbc/lib/libsqlod.so

Description = OSDL-DBT-1

is a valid .odbc.ini file where dbt1 can be used as the server name to connect to the database.

4Running the Test Kit

The script, run_test_sample.sh, in the scripts/ directory is provided as a sample on how to run the test kit on a single system. The variables that should be changed are:

5Test Results

The driver program, in both its forms, records response time data to the file mix.log. A program, results, is provided in the tools/ directory, which takes the file name as a command line argument to analyze the file. The results program also assumes the output files from the cpu.sh, io.sh, and paging.sh are in the scripts/stats/ directory relative to the tools/ directory. The results program displays the results of these files to the screen, for example:

interaction % avg response time (s)

AC 0.04 0.000

AR 0.05 0.000

BS 5.19 0.000

BC 1.08 0.000

BR 2.50 0.000

CR 2.92 0.000

HO 17.19 0.000

NP 4.98 0.000

OD 0.63 0.000

OI 0.78 0.000

PD 16.71 0.000

SR 19.76 0.000

SU 16.55 0.000

SC 11.62 0.000



12.6 interactions per second

17.7 minute duration



CPU:

Average %user 11.43

Average %nice 0.00

Average %sys 7.26

Average %idle 81.30



I/O:

The following numbers are presented as averages for each column.

Note that stats are reported for every device attached to the system.

[Note: The following lines have wrapped in this document.]

device rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util

hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.24 0.00

16.77 10.50 0.00

hda2 7.38 4.25 2.29 2.62 77.39 55.23 27.89 1.25

360.66 181.54 6.34

hda3 0.17 0.00 0.02 0.00 1.59 0.00 11.87 0.00

39.44 26.33 0.03

hdb 0.00 0.00 0.01 0.01 0.10 0.09 2.29 0.00

46.27 3.41 0.00

hdb1 0.00 0.00 0.01 0.01 0.10 0.09 2.29 0.00

46.27 3.41 0.00

hda 7.73 4.25 2.37 2.67 80.82 55.65 27.72 1.29

364.82 183.28 6.50



Paging:

Average pgpgin/s 43.67

Average pgpgout/s 27.91

Average activepg 29601.77

Average inadtypg 14288.26

Average inaclnpg 2696.00

Average inatarpg 2112.82

6Advanced Topics

6.1Additional Compiler Options

The following section describes the compiler flags defined in common.mk that can be used to generate different binaries from the default:



For example, in common.mk, the following CFLAG generates the appServer which generates time.log and sends SearchResults on title and SearchResults on author to the cache module.

CFLAGS = -g -I$(INCLUDE_DIR) -I$(SAPDBROOT)/incl -DGET_TIME -DSEARCH_RESULTS_CACHE

6.2Manually Running the Test Kit

The OSDL-DBT-1 test kit can be run in two ways. The kit is designed to be run in a two-tier and a three-tier environment. The two-tier environment involves running the driver against the database directly while the three-tier environment involves running the driver against a type of transaction management program on the middle-tier system that sits in front of the database. A sample script it provided to run the kit in a three tier environment. Running the kit manually for each environment is explained later in this section.

6.2.1Two-Tier Environment

Frame3

The driver program for the two-tier environment is compiled as dbdriver_p1 in the dbdriver/ directory. The dbdriver_p1 program takes the following arguments in the following order:

6.2.2Three-Tier Environment

Frame4

If tests are conducted in a three-tier environment, then the applications that are started are:



The appServer program must be started before the driver in the three-tier environment. The program resides in the appServer/ directory and take the following arguments in the following order:



The driver program for the three-tier environment is compiled as dbdriver_p2 in the dbdriver/ directory. The dbdriver_p2 program takes the following arguments in the following order:

connect.

6.2.3Three-Tier Environment with Cache



Frame2If tests are conducted in a three-tier environment then the applications that are started are:



Since SearchResults on title and SearchResults on author interactions do not need to reflect any changes in the database, a cache module is provided to serve those two interactions. When the cache module starts, it accesses the database to warm up the cache. Then it waits for connections from the appServer.



The cache program must be started before the appServer. The program resides in the cache/ directory and take the following arguments in the following order:



The appServer program must be started before the driver. The program resides in the appServer/ directory and take the following arguments in the following order:



The driver program for the three-tier environment is compiled as dbdriver_p2 in the dbdriver/ directory. The dbdriver_p2 program takes the following arguments in the following order:

connect.

6.3Tools

Various tools are provided in the test kit that aid in setting up the database or testing different functions of the kit. These tools are covered in this section.

6.3.1ODBC Test Tools

The tools explained in this section are: load_db_proc, odbc_test_p1, and odbc_test_p2. Database specific tools are listed under the appropriate sections.



The odbc_test_p1 and odbc_test_p2 programs , in the tools/ directory, are used to support isolated testing of the database stored procedures. And load_db_proc is the program to load long stored procedures that repmcli cannot load. Due to limitations in repmcli , the program fails to load a stored procedure if the size of the stored procedures is too large. The load_db_proc program is provided as a tool to load those zlong stored procedures. To execute load_db_proc , go to the tools/ directory and type:

./load_db_proc <servername> <uname> <password> <db_procedure_name>

where:



The odbc_test_p1 program is the tool used to test each individual interaction. It connects to the database directly. To execute the program, go to directory tools and type:

./odbc_test_p1 <interaction> <servername> <uname> <password>

The <interaction> option can be one of the following:



The <servername> is the database node name in the format of hostname:database_name. For example, localhost:DBT1.



The <uname> option is the database user name: dbt.



The <password> options is the database user password: dbt.



The odbc_test_p2 program is used to test each individual interaction and it connects to the appServer program instead of connecting directly to the database. To execute this program, go to directory tools and type:

./odbc_test_p2 <interaction> <servername> <port>

The interaction options are the same as those for the odbc_test_p1 program. The <interaction> option can be one of the following:



The <servername> option is the hostname or IP address of the system where the appServer program is running on.



The <port> option is the port number where the appServer is listening to. It is the port number used when starting the appServer program.

6.3.2System Resources

The test kit provides scripts to help collect processor, disk, and memory utilization, using iostat, in the scrips/stats/ directory, cpu.sh, io.sh, and paging.sh respectively. Each script takes two command line arguments, in the following order, the sample length in seconds, and the number of samples to take. To collect ten minutes of processor utilization data over 30 second intervals:

cpu.sh 30 20

6.3.3Database Notes

6.3.3.1SAP DB

The tools provided for SAP DB are a set of scripts that aid in using the database.



The backup_db.sh script backups the database.



The drop_db.sh script drops the database.



The monitor.sh script takes a snapshot of all of the counters in most of the monitor tables and outputs them into text files.



The restart.sh script shutdown down and start the database.

The trace_off.sh script shutsturns off the database trace and takes a command line argument for where to output the trace data. For example:

trace_off.sh /tmp/todays_trace.txt

Be sure to run trace_on.sh first, or else there will not be any trace data collected.



The trace_on.sh script turns on the database trace.

The update_stats.sh script updates the statistics table in the database to help the optimizer.



Within SAP there exist internal variables that you may wish to change. Following is a description and an example of how to change them.



DATA_CACHE - The database cache.

MAXUSERTASKS - The maximum amount of connections a single user can have to the database at any one time. This is set to 50 by default.

MAXCPU - The number of CPUs in your system (default is 1). This must be changed if you have a smp machine to achive better results. The database will not use more then the number of processors defined here.



To change MAXCPU on an 8-way system:

dbmcli -d DBT1 -u dbm,dbm

param_startsession

param_put MAXCPU 8

param_checkall

param_commitsession

db_restart



See the SAP documentation at http://www.sap.org/ for more information.

16