OSDL-DBT-1
An On-Line Transaction Processing Test Kit
Similarities and Differences to the TPC-W™
Version 1.0
Table of Contents
1Similarities and Differences to the TPC-W™ Specification 3
1.1Components 3
1.2Definition of Terms 3
1.2.1RBE vs. Driver 3
1.2.2EB vs. EU 3
1.2.3Web Interaction vs. Interaction 3
1.2.4Transaction Management Server 4
1.3Implementation Rules 4
1.3.1The Driver 4
1.3.2 The Database 5
1.3.2.1Database Entities, Relationships, and Characteristics 5
1.3.2.2Scaling and Database Population 5
1.3.2.3ACID Properties 5
1.3.2.4Integrity Rules 6
1.3.2.5Checkpoints 6
1.3.2.6Isolation Level 6
1.3.2.7180-Day Space Requirement 6
1.3.2.8Foreign Keys 6
1.3.2.9Stored Procedure 7
1.4The TPC 8
TPC-W™ is composed of a Remote Browser Emulator (RBE), a Payment Gateway Emulator (PGE) and the System Under Test (SUT). The SUT includes network components, web servers, application servers and a database server.
OSDL-DBT-1 has only a driver, a transaction management server and a database server. The is no PGE used to simulate credit card authorizations.
Some terms defined in the TPC-W™ do not apply to the OSDL-DBT-1 test kit and some have different meanings. The TPC-W™ terms will be used if they apply to both cases. Where terms differ, they will be defined for the OSDL-DBT-1 test kit.
The Remote Browser Emulator (RBE) is the software component that drives the TPC-W™ workloads.1 It creates and manages the Emulated Browsers (EB). The OSDL-DBT-1 driver controls the DBT-1 workloads. It creates and manages the Emulated Users (EU).
The Emulated Browser (EB) refers to the entity (e.g. a process or thread) that emulates a user communicating via a browser for HTML content. Packets are sent through HTTP and TCP/IP protocols over a network connection to the SUT.2 An Emulated User refers to the entity (e.g. a thread) communicating with the transaction management server by sending request input and receiving output data via TCP/IP sockets and thus eliminates the need for HTTP servers in the kit.
The term web interaction in the TPC-W™ benchmark refers to a complete cycle of communication between the EB and the SUT. The exchange of messages may include HTTP requests, cookies, HTML pages, image files or other web objects.3 There are no web objects in the OSDL-DBT-1 test. An interaction refers to a cycle of communication between an EU and the transaction management server. The exchange of messages includes request input and output data.
While the TPC-W™ benchmark requires web servers to serve web objects, the OSDL-DBT-1 test kit contains a transaction management server that performs the transaction management functions.
In the OSDL-DBT-1 test kit, the transaction management server is the middle tier connecting the driver and the database. It performs transaction management by creating and managing database connections and a transaction queue.
Web logging4 does not apply to the OSDL-DBT-1 test.
Other terms, such as "user session", "customer", "shopping session", "database transaction", "randomly selected with [x..y]", and "non-uniform random function (NURand)", are the same in both cases. Refer to the TPC-W™ Specification Clause 2.1 for the definition of these terms.
The driver has the same responsibilities as the RBE5, except for providing security for some web interactions. In the TPC-W™ benchmark, the secure web interactions are: Buy Request, Buy Confirm, Order Inquiry, and Order Display. In secure web interactions, all communications between the EB and the SUT must be encrypted and communicated with SSL version 3 or with TLS using SSL_RSA_WITHRC4_128_MD5 as the cipher suite.6 The OSDL-DBT-1 driver does not provide any security mechanisms.
Like the TPC-W™ benchmark, the OSDL-DBT-1 driver uses TCP/IP sockets for all network communication.7
However, the TPC-W™ Specification requires that a new socket must be created for each user session and user sessions can not share a socket connection.8 The OSDL-DBT-1 driver creates a socket connection for each EU, and keeps that socket open for the duration of the test.
The EU sends input following the rules defined in the TPC-W™ Specification Clause 2.3 "Input Requirements" with some exceptions. Due to limitations in the SAP DB, some of the stored procedures have the number of input and output parameters reduced. For a description of stored procedure limitations, refer to Section 1.3.2.9.
In the TPC-W™ benchmark, each web interaction results in the presentation of a page to the EB.9 After receiving the page, the EB parses that page, and selects the web interaction to be requested next. In the OSDL-DBT-1 test, each interaction results in the output data being sent directly to the EU. After receiving the data, the EU selects the next interaction to request. The EU chooses the navigation options in the same way as the EB.10 The driver maintains the interaction mix defined in the TPC-W™ Specification Clause 5.2.1.
After each interaction, the EU simulates a user thinking (think time) by sleeping for a certain period of time. Think time is generated randomly according to the TPC-W™ Specification Clause 5.3.
The EU ends a user session when the conditions defined in the TPC-W™ Specification Clause 6.2.2.1 are met or when the duration of the test expires. The TPC-W™ Specification requires the EBs to close the network connection to the web servers when the user session ends, and start a new session for a new EB.11 The OSDL-DBT-1 driver keeps a socket open for the duration of the test.
The OSDL-DBT-1 database consists of 8 individual base tables as defined in the TPC-W™ Specification Clause 1.3.1. Table layouts are defined in the TPC-W™ Specification Clause 1.4.
The OSDL-DBT-1 data generator mostly follows the guidelines in the TPC-W™ Specification. The OSDL-DBT-1 data generator generates dates starting from the year 1900 while the TPC-W™ data generator guidelines requires generating dates starting from the year 1800.12 The OSDL-DBT-1 database follows the scaling rules defined in the TPC-W™ Specification Clause 4.1, 4.2 and 4.3 and its database population follows the rules defined in the TPC-W™ Specification Clause 4.1.
No clustering, partitioning, or replication is used in the OSDL-DBT-1 database.
The current version of the OSDL-DBT-1 test kit does not contain any ACID tests defined in the TPC-W™ Specification Clause 3.1.
The current version of the OSDL-DBT-1 test does not enforce the integrity rules as defined in the TPC-W™ Specification Clause 1.7.
The current version of the OSDL-DBT-1 test kit does not perform checkpoints as defined in the TPC-W™ Specification Clause 5.5.3.
The current version of the OSDL-DBT-1 test uses the default isolation level set provided by the SAP DB.
In the SAP DB, the default isolation level is 1.13 With this isolation level, dirty reads are not possible, and non-repeatable reads as well as phantom phenomenon can occur.14
The TPC-W™ Specification requires a Level 3 isolation level between any two TPC-W™ update transactions, and a Level 2 isolation between any TPC-W™ read-only transaction and a TPC-W™ arbitrary transaction.15 This rule does not apply to three transactions: New Products, Best Sellers, and Admin Confirm. The database reads to generate the list of 50 New Products/Best Sellers items may be performed at an isolation level of 0. Admin Confirm requires the database to generate five item IDs; this can be performed at an isolation level of 0.16
For the definitions of dirty reads, non-repeatable reads, phantom phenomenon, TPC-W™ update transactions, TPC-W™ read-only transactions, and TPC-W™ arbitrary transactions, please refer to the TPC-W™ Specification Clause 3.1.4.1.
The current version of the OSDL-DBT-1 test kit does not follow the 180-day space computation rules defined in the TPC-W™ Specification Clause 4.4.
The current version of the OSDL-DBT-1 test kit does not implement the foreign keys as defined in the TPC-W™ Specification Clause 1.4.
Stored procedures are created to execute the interactions. Stored procedures accomplish the processes defined in the "Processing Definition"17 sub sub-sections for each interaction. In the TPC-W™ Specification, a response page is defined for each web interaction. The OSDL-DBT-1 stored procedures return the same data required in the response page section with some exceptions.
Due to limitations in the SAP DB, stored procedures can not always return the required information defined in the TPC-W™ Specification.
Shopping Cart: The TPC-W™ Specification Clause 2.4.2 requires a list of I_ID, QTY pairs as input; the number of pairs can be up to 100. Clause 2.4.3.5 requires that this stored procedure returns item information in the CART. There can be up to 100 items. Due to the length limitation of the stored procedures in the SAP DB, the number of input and output items is reduced to 20.
Order Display: The TPC-W™ Specification Clause 2.9.4 requires that this stored procedure returns order line information in an order. There can be up to 100 order line items. Due to the length limitation of the stored procedures in the SAP DB, the number of output order items is reduced to 20.
Buy Request: The TPC-W™ Specification Clause 2.6.4 requires that this stored procedure returns item information in the CART. There can be up to 100 items. Due to the length limitation of the stored procedures in the SAP DB, the number of output items is reduced to 20.
Buy Confirm: The TPC-W™ Specification Clause 2.7.4 requires that this stored procedure returns item information in the CART. There can be up to 100 items. Due to the length limitation of the stored procedures in the SAP DB, the number of output items is reduced to 20. Furthermore, Buy Confirm does not obtain the credit card authorizations from the PGE.
Best Sellers: This complicated query requires cursor support in the stored procedure. Unfortunately, the SAP DB stored procedures do not have cursors implemented. Therefore, the current OSDL-DBT-1 Best Sellers stored procedure executes New Products.
Admin Confirm: This complicated query requires cursor support in the stored procedure. Unfortunately, the SAP DB stored procedures do not have cursors implemented. Thus, the current OSDL-DBT-1 Admin Confirm stored procedure only updates I_COST, I_IMAGE and I_THUMBNAIL and returns the item information.
The OSDL-DBT-1 data analysis tool analyzes some performance information (e.g. the percentage of each interaction executed during the measured interval, average interaction response time, and number of interactions executed per second). The OSDL-DBT-1 results are not comparable to the TPC-W™ results. The performance data differences are noted below:
Instead of WIPS18, OSDL-DBT-1 test reports interactions per second.
Instead of WIRT19, OSDL-DBT-1 test reports interaction response time.
The current version of the OSDL-DBT-1 test collects but does not report the average duration of the think time20 over the measured interval.
The OSDL-DBT-1 test does not report the 90% percentile of the interaction response times.
The OSDL-DBT-1 test does not report the throughput rating21.
The OSDL-DBT-1 test does not enforce the measurement interval requirements defined in the TPC-W™ Specification Clause 5.5. It is up to the users to decide the measurement interval duration.
The OSDL-DBT-1 test reports CPU utilization, I/O activity and page/swap activity of the database machine. It does not report the other performance statistics required in the TPC-W™ Specification Clause 5.7.2.
The TPC is a non-profit corporation founded to define transaction processing and database benchmarks and to disseminate objective, verifiable TPC performance data to the industry. TPC and TPC Benchmark W are trademarks of the Transaction Processing Performance Council. For more information about the TPC please contact info@tpc.org or visit www.tpc.org.
1 TPC-W™ Specification Version 1.7 Clause 6.1.1.
2 TPC-W™ Specification Version 1.7 Clause 2.1.3.
3 TPC-W™ Specification Version 1.7 Clause 2.1.4.
4 TPC-W™ Specification Version 1.7 Clause 2.1.8.
5 TPC-W™ Specification Version 1.7 Clause 6.1.4.
6 TPC-W™ Specification Version 1.7 Clause 2.2.1.
7 TPC-W™ Specification Version 1.7 Clause 6.1.6 and 6.1.7.
8 TPC-W™W™ Specification Version 1.7 Clause 6.1.7.
9 TPC-W™ Specification Version 1.7 Clause 5.2.2.2.
10 TPC-W™ Specification Version 1.7 Clause 5.2.2.3.
11 TPC-W™ Specification Version 1.7 Clause 6.2.2.2.
12 TPC-W™ Specification Version 1.7 Clause 4.7.1.
13 Reference Manual: SAP DB P 211
14 Reference Manual: SAP DB P 212
15 TPC-W™ Specification Version 1.7 Clause 3.1.4.
16 TPC-W™ Specification Version 1.7 Clause 2.12.3.1, 2.13.3.2 and 2.16.3.3.
17 TPC-W™ Specification Clause 2.3.3, 2.4.3, 2.5.3, 2.6.3, 2.7.3, 2.8.3, 2.9.3, 2.10.3, 2.11.3, 2.12.3, 2.13.3, 2.14.3, 2.15.3, and 2.16.3.
18 TPC-W™ Specification Version 1.7 Clause 5.7.1.
19 TPC-W™ Specification Version 1.7 Clause 5.1.5.
20 TPC-W™ Specification Version 1.7 Clause 5.3.1.2.
21 TPC-W™ Specification Version 1.7 Clause 5.4 and Clause 5.6.2.