Mates manual

This is the manual for Manual Tester (Mates) v. 1.2.

Content

1 Mates concepts
2 Installation
3 Writing the test scripts
4 User guide - Administrator
5 User guide - Test manager
6 User guide - Tester
7 User guide - Programmer
8 User guide - Support
9 User guide - Common functionality
10 Production notes

1 Mates concepts

1.1 Overview

Mates is an information system to aid manual testing according to test scripts.

The test scripts are created in MS Excel and saved as XML. Then a provided tool is used by Mates test manager to merge a selected subset of tests into a single XML file, which is uploaded into Mates. Mates parses the file, extracts the included test scripts and creates corresponding records in its database.

The testers then login into Mates and fill in the test scripts during the testing. When finished, the testers mark the scripts as passed or failed.

The testers can also raise issues encountered during testing. The issues reported by testers are consolidated and confirmed by Mates test manager. All the confirmed issues are then resolved by the programmers. Finally, the resolved issues are subject to re-test by the test manager to verify that they were fixed.

In any moment, anyone can review the current, up-to-the-second status of the testing - numbers of test scripts that are passed, failed, or being worked on, number of issues that are reported, confirmed, or resolved, etc.

Mates also implements issue discussion, support request management, a standard, role-based user access system, and logging of user actions including scheduled purge of old log entries.

1.2 Test run

Tests are performed in test runs. Usually, at first all the test scripts are done. That uncovers some bugs in the system tested. After the bugs are believed to be fixed, a second run of the test scripts is done, this time perhaps only the scripts which were failed in the first run. Finally, a third test run should be done, including all the test scripts again to make sure all bugs are eliminated and system works perfectly.

In addition, each of the three test runs described above can be also split into two, first including "smoke" tests, second including the detailed tests of the tested system. The smoke tests are simple, high-level overview tests that verify testability of the system. If any of the smoke tests fail, the testing is stopped - no detailed test scripts are executed until all the smoke tests pass.

In Mates, test scripts are organized in test runs. Every test script belongs to a specific test run. A test run is a wrapper for the test scripts. A test run must be created to upload test scripts into Mates. Additional test scripts can be subsequently uploaded into an open test run. A test run can be closed after all contained test scripts are closed (either passed or failed). There can be multiple test runs open at the same time.

Test run state chart

1.3 Test script

A test script is uploaded into status Available. Once a tester starts working on the script by updating its content, it becomes Open. An Open script cannot be edited by any other tester. When finished with the script, the tester can make it Pass or Fail. Once passed or failed, the tester cannot modify the script anymore.

Script state chart

The test manager can reopen a passed of failed script, and can also cancel assignment of a script from a tester.

1.4 Component

Each issue is assigned to a single component. The test manager defines the components before the testing so that issues can be assigned to them.

Components primarily help differentiate the responsibility for fixing the bugs, so it is recommended that the components follow the separate development teams involved in the project. Alternatively, they can be defined on the level of the individual team members to track responsibility for fixing a particular bug to a single programmer.

NOTE! It is recommended that one of the components be "Test script". This way any potential problems in the test scripts themselves can be tracked in Mates and not forgotten to be fixed before the next test run.

1.5 Version

Each reported/confirmed issue is assigned to a single version of the tested system. This indicates in which version of the system the issue has been found.

Each fixed issue is assigned to another version of the system, documenting the version containing the fix.

The test manager defines the versions before the testing so that issues can be assigned to them.

1.6 Issue

During testing, the testers can report issues (potential bugs, problems encountered in the tested system). Issues are created in status Reported.

The issues reported by testers are then reviewed by Mates test manager - he can re-word them, and change their severity and component assignment. Once updated, the test manager confirms the issues. Issues become Confirmed. At this stage, the test manager can also resolve duplicities (same issues reported by multiple testers), invalid issues, etc.

All the confirmed issues are then resolved by the programmers. Issues become Resolved.

Finally, the resolved issues are subject to re-test by the test manager to verify that they were fixed. Issues become Verified.

Test manager can re-confirm an issue from any status, i.e. also from Resolved (issue was not really resolved), and Verified (regression).

Issue state chart

1.7 Support request

The testers can raise support requests to be helped with the execution of a test script. E.g. interface between two information systems is tested, the testers access one of the systems, but cannot access the other system directly - they have to ask somebody else to perform the tasks in that other system.

The tester creates the support request. Its status becomes Pending.

The support personnel perform the tasks requested, and mark the request as Fulfilled. The tester can then review the comments by the support personnel and continue with the execution of the test script.

Support request state chart

1.8 Data model

A high-level data model is depicted below.

Data model

1.9 User roles overview

The following table presents an overview of Mates user roles and their primary responsibilities.
Role Primary responsibilities
Test manager Maintain components.
Maintain versions.
Create test runs.
Upload test scripts into Mates.
Close test runs.
Confirm issues.
Verify issues.
Tester Perform tests execution.
Report issues.
Create support requests.
Programmer Resolve issues.
Support Fulfill support requests.
Administrator Maintain users.
Maintain user roles.
Maintain system processes.
Review [none]

2 Installation

2.1 Java JDK

Download Java JDK from
http://java.sun.com. Mates is currently using JDK v. 1.4.2 and 1.5.0.

NOTE! Java JRE is not sufficient. JBoss requires JDK to compile JSP pages.

2.2 Apache Ant

Download Apache Ant from http://ant.apache.org and install it. Ant is used by the script processor tool.

2.3 JBoss

Download JBoss application server from http://www.jboss.com. The supported versions are 3.2.x and 4.x. Mates is currently developed on version 3.2.7 and 4.0.3 SP 1.

Install the JBoss application server properly - follow the JBoss documentation. Generally it is sufficient to set JAVA_HOME system property to the path where the Java JDK has been installed, unzip JBoss, and simply run it.

2.4 Mates

Download latest Mates version from Mates download.

Extract the downloaded zip file into a chosen directory. The directory structure of the unzipped file is presented in the table below.
Resource Content
app Development only. Contains the Mates definition XML file in SLIS (Small Language of Information Systems) - mates.xml. Contains all the externalized strings of Mates application - file matesLang.properties.
build Contains the Java class files of the Mates tools used by the script processor tool.
dist Contains the Mates distribution file - mates.ear.
doc Contains the Mates documentation.
functional Development only. Contains the automated Mates functional tests. The tests are automated using Canoo WebTest and DBUnit.
resources Development only. Contains the HTML resources of Mates application.
script Contains the script processor tool.
build.xml Development only. This is the SEAF build file for the Mates application.
src.zip Development only. Contains the Mates POJO model classes implementing its business logic in the SEAF framework.

Copy the <mates_home>/dist/mates.ear file into the hot deploy directory of JBoss - <jboss_home>/server/default/deploy.

Optionally, if you are running JBoss in a console, watch deployment progress until it says that mates.ear is successfully deployed.

Now Mates is running under JBoss. Launch a web browser and point it to http://<hostname>:8080/mates/ (if your browser runs on the same machine as the JBoss server, use http://localhost:8080/mates/). When the page loads, you should see "Manual Tester home page" heading and a link "Enter the application..." underneath it.

Now the Mates installation has been successfully accomplished.

NOTE! This installation procedure assumes that JBoss has been installed as-provided and no additional configuration was done. In such a case, JBoss takes care of the deployment automatically: e.g. it creates the required DB tables in its default Hypersonic database, etc. If you deploy Mates on a JBoss installation with has been altered, please contact your JBoss administrator to help you with Mates deployment.

3 Writing the test scripts

3.1 Configuring the script template

Open MatesTemplate.xml file (found in the <mates_home>/doc directory) in MS Excel (right-click on the file, select Open with..., Microsoft Excel). Unhide columns 7 to 13.

Define Domains in columns 10, 11. The domains should reflect functional areas in the tested system and thus categorize the test scripts. E.g. in Mates itself, the domains would be Test run, Script, Issue, etc.; in an ERP system the domains could be Product catalogue, Purchase Order, Delivery Note, Receiving Note, Invoice, etc.

Optionally modify Categories in column 13.

If you have changed the number of Domains (or Categories), select the Domain field B2 (or Category field B5), invoke menu Data - Validation, and update the Source accordingly.

Hide columns 7 to 13 again and save the template file.

3.2 Writing a new script

To write a test script, copy the MatesTemplate.xml file into a file of the test script (e.g. PO100.xml), and open the copied (PO100.xml) file in MS Excel (right-click on the file, select Open with..., Microsoft Excel).

Now fill-in the test script header. Choose Domain from the list (e.g. PO), enter the Code (e.g. 100) and Name (e.g. Create Purchase Order). Choose the Category (e.g. Core). Enter the Description and Prerequisites.

Now enter the steps of the test script. For each step enter the Task (e.g. Choose a supplier), Result name (e.g. Supplier number), and choose Result type from the list.

The result types supported by Mates are:

When finished with the steps, delete the remaining numbered rows.

Now save the file. You have just written a Mates test script.

NOTE! The scripts are uniquely identified by a combination of the Domain and Code attributes. Any other attribute can change, but the script identity remains the same.

NOTE! Use Version to distinguish between different versions of the test script. Each time you modify a script, change its version. Keep all historical versions of the test scripts for reference.

NOTE! The blue colored fields at the right are included to support optional printing of the scripts and filling them on paper, as a fallback procedure. (Not that we had ever to use it...)

NOTE! Mates expects to receive the test scripts in UTF-8 character coding. Take care to avoid misinterpreting characters because of the encoding. From experience this can happen e.g. when MS Word is used to write the test script steps: (depending on setting) it replaces straight quotes with rounded quotes, which then cause problems when uploading to Mates, similarly replacing minus with long dash by MS Word causes the same problems.

NOTE! Review the PO100.xml and PO101.xml files in the <mates_home>/doc/example directory, they contain examples of created Mates test scripts. (Do not forget to open them in Excel.)

3.3 Creating a test suite

To upload test scripts into Mates, a test suite file has to be created first. Choose the test scripts to be uploaded to Mates. Copy the corresponding test script XML files to <mates_home>/script/excel directory. Go to <mates_home>/script/processor directory and type ant on command line. After the ant execution completes successfully, go to to <mates_home>/script directory. Here the test suite files have been created: The content of the TestSuite.xml file can now be uploaded to Mates application, see the corresponding sections on creating Test run and uploading test scripts in the Test manager's User guide chapter.

NOTE! The TestSuite.html overview can be produced by the script processor tool even without uploading test scripts to Mates. It is the best suited form e.g. for business people to review the test scripts before the start of the testing.

NOTE! Review the TestSuite.xml and TestSuite.html files in the <mates_home>/doc/example directory, they contain examples of generated Mates test suite files.

4 User guide - Administrator

4.1 First use - system setup before testing

Create the root user

Enter the following URL into your browser manually: http://<hostname>:8080/mates/user.root.detail.do

The User root detail screen is displayed. Enter "root" into the LoginName field and "password" into the Password field, and press "Create root" button.

The system displays confirmation message "The record has been created." Now follow the link "Log in..." in the top right corner of the browser to open the login screen, enter "root" into the LoginName field and "password" into the Password field, and press "Login" button.

Now the home screen should be displayed, and the "Logged in as: root" message should be indicated in the top right corner of the browser.

Create user roles

While logged in as root, go to System - Role search. Press button RecreateRoles. This creates the default Mates user roles (Administrator, Test manager, Tester, Programmer, Support, Review) together with their default access rights.

To review / modify role access rights (permissions), select the role on the Role search screen and go to screen Role permissions. For the selected role, a list of assigned permissions and a list of unassigned permissions are displayed. The assigned permissions list includes screen codes of screens accessible by a user with the role. The unassigned permissions list includes screen codes of screens which cannot be accessed by such user (with some exceptions, e.g. the login screen). To enable unassigned screen, click on its code - the code is moved from unassigned list to the assigned list. To disable assigned screen, click on its code - the code is moved from assigned list to the unassigned list.

You can recreate the user roles anytime in the future. This will restore the default roles with their default rights. However the users will have to be assigned to the roles after that again.

Create users

While logged in as root, go to System - Create user. Fill in the fields with values of a new user (e.g. LoginName admin, etc.) and press Create. Now, go to the User roles screen, and from the list of unassigned roles select the roles that the user should have (e.g. Administrator).

After you have created an Administrator account in the system, login again as Administrator, and perform all the following administration tasks under the Administrator account.

NOTE! Do not use the system from the root user. Create an administrator account to work with the system.

NOTE! Do not assign multiple roles to a single user unless actually required. It is recommended that each user has a single role. It is however possible to assign multiple roles to a single user.

Create processes

While logged in as Administrator, go to System - Process search. Press button RecreateProcesses.

If required, you can now start the Logger process: follow the Logger process name and press button Start. This begins creating log entries that can be reviewed at the System - Log screen.

If you intend to keep the logger on, it is recommended that you schedule log purging to prevent database overflow. On the System - Process search screen follow the PurgeLog process name. In the Process schedule list section select Hour 2 and Minute 00 and press Update. Then press EnableScheduling. This setting will purge all log records older than 1 month every day at 2 a.m.

NOTE! If using log, make sure the log purging is scheduled properly to avoid database overflow.

Now log in as a Test manager and continue with system setup in section 5.1.

5 User guide - Test manager

5.1 System setup before testing

Components

Login as Test manager and go to Project - Components. Enter the component Name and press Create for each component of the tested system. See Mates concepts chapter for general info on components. Create a component "Test script" for reporting errors in the test scripts themselves.

Versions

Go to Project - Versions. Enter the version Name (e.g. 0.0.0) and press Create for each version of the tested system. See Mates concepts chapter for general info on versions.

The Mates system is now prepared to be routinely used. See the following section on how to load test scripts into the application.

5.2 Test run

Create test run

Go to Test run - Create test run. Enter the Name (e.g. "1 Smoke"). Open the TestSuite.xml file generated by the script processor in any viewer (e.g. Notepad) and copy its content via clipboard into the TestSuite field. Press Create. The system displays message "The record has been created."

Now the testers can log into Mates application and start filling the test scripts.

NOTE! If you have access to JBoss console or JBoss system.log file you can review the progress of processing the test scripts of the test suite file.

Uploading additional test scripts

It is possible to upload additional test scripts for an open test run.

Go to Test run - Test run list and select the test run by clicking on its name. The Test run detail screen is displayed. Now go to Test run - Upload tests, copy the content of the TestSuite.xml file with the additional scripts into field TestSuite and press Upload.

Mates validates and processes the uploaded test scripts as follows:

If the upload concludes successfully, the system displays message "The record has been created."

NOTE! When the upload is aborted due to the fact that an open test script was included in the test suite file, remove the script from the <mates_home>/script/excel directory and re-generate the test suite file.

Close test run

Go to Test run - Test run list. Click on the Open test run Name. Click button Close to close it.

NOTE! A test run can only be closed if all its test scripts are closed (Pass or Fail). Finish all Open and Available test scripts before closing the test run.

5.3 Test script

Reopen test script

When a script has been closed by mistake or requires updates after being closed, the Test manager can reopen it.

Go to Script - All scripts, find the script (in status Pass or Fail) and click on its Name. Go to Maintain script and press button Reopen. The script status becomes Open.

Cancel assignment of a test script

When a script has been taken by a tester, and needs to be processed by another tester, the Test manager can cancel its assignment.

Go to Script - All scripts, find the script (in status Open) and click on its Name. Go to Maintain script and press button CancelAssignment. The script status becomes Available.

5.4 Issue

Create issue

It is not expected that Test manager would normally create issues, rather the testers should be doing that, however that option is available.

Go to Issue - Create issue, enter the issue attributes, and press Create. The issue is created in status Reported.

NOTE! An issue created in this way is not related to any test script and test step. This is suitable only for general issues that cannot be associated with any test script.

Confirm reported issues

Go to Issue - Issue search, set selection criteria Status to Reported and press Refresh. All the issues newly reported by testers are displayed. Click on an issue Id, the Issue detail screen is displayed. Review (and update, if needed) the data entered by the tester and press Confirm. The issue status becomes Confirmed. Go to Issue search screen again and continue with the next Reported issue.

Eliminate wrong issues

Test manager can resolve issues. This is mostly used to resolve issues as invalid, duplicate or such that will not be fixed.

Go to Issue - Issue search, find the issue and click on its Id. Go to Resolve issue. Choose issue Resolution:

Enter the ResolverComment explaining why the issue has not been fixed and press Resolve. The issue status becomes Resolved.

NOTE! When resolving issue as either Invalid, Wontfix or Duplicate, the ResolverComment field must be filled.

NOTE! It is not expected that Test manager will resolve issues as Fixed. This is the job of the Programmers.

Verify resolved issues

Go to Issue - Issue search, set selection criteria Status to Resolved and press Refresh. All the resolved issues are displayed. Click on an issue Id, the Issue detail screen is displayed. Review (and update, if needed) the issue attributes.

Now perform the retest on the tested system to verify that the issue has actually been resolved as indicated by the programmer.

If the re-test succeeds, enter the Verificaion Reference and press button Verify. The Verification Reference is e.g. a purchase order number on the test system which has been used for the verification re-test. The issue status becomes Verified. Go to Issue search screen again and continue with the next Resolved issue.

NOTE! If the retest fails, you have the option to re-confirm the issue again, if required - just press button Confirm and the issue status becomes Confirmed.

5.5 Support request

Fulfill pending support request

It is not expected that Test manager would normally fulfill support requests, rather the Support personnel should be doing that, however that option is available.

Go to Support - Support request search, find the request and click on its Id. Go to Support request detail screen, enter the FulfillmentInfo field and press Done. The request status becomes Fulfilled.

6 User guide - Tester

6.1 Test script

Take an available script

Go to Script - Available scripts. Choose a script that you want to execute and click on its Name, the Script detail screen is displayed.

Press button Update. The script is set to status Open, and the TesterName and StartAt attributes are filled-in automatically by the system.

The script is from now not visible from the Available scripts screen, so that no other tester can take it. You, as its owner, can however see it in the My scripts screen.

Fill in script

To select a script you have already taken, go to My scripts, find the script and click on its Name.

Now you are on the Script detail screen and you can fill in the script steps, specifically the ResultValue column. There are 3 types of ResultTypes:

You can save the entered values to database at any time by pressing the Update button.

You can make the script passed by pressing the Pass button. The script must have all the steps filled in to make it passed. The script status becomes Pass and its EndAt attribute is filled in by the system.

You can make the script failed by pressing the Fail button at any time. The script status becomes Fail and its EndAt attribute is filled in by the system.

You can create an issue for a specific step by clicking on the Issue link next to the step. See Create issue section on how to fill in the values on the Create issue screen.

You can create a support request for a specific step by clicking on the Support link next to the step. See Create support request section on how to fill in the values on the Create support request screen.

6.2 Issue

Create issue

When a problem arises during executing a step of a test script, click the Issue link next to the step. This brings you to the Create issue screen.

Fill in the accurate and exact description of the problem into the Summary field. Select the Component and system Version in which the issue has been found. Select the issue Severity. Press the Create button.

The issue has now been created.

Now return to Script - Script detail screen and continue with filling in the script.

NOTE! An issue can also be created without a link on a script step, just by going to Issue - Create issue screen. However it is highly recommended to always create issue with a link to a particular script step to enable reproducibility of the issue by its future resolver.

6.3 Support request

Create support request

When you need someone else to do something required by the script step that you cannot do, click the Support link next to the step. This brings you to the Create support request screen.

Fill in the accurate and exact description of what you need to be done into the Todo field and press the Create button.

The support request has now been created.

You can now continue with another script, or wait until the support request gets fulfilled by a Support colleague by refreshing the Support - Support request search screen with the support request displayed until its status changes from Pending to Fulfilled.

When the support request becomes Fulfilled, click on its Id and on the Support request detail screen review the FulfillmentInfo field.

NOTE! A support request can also be created without a link on a script step, just by going to Support - Create support request screen. However it is highly recommended to always create support request with a link to a particular script step to provide a helpful context for the Support personnel when fulfilling the request.

7 User guide - Programmer

7.1 Issue

Create issue

It is not expected that Programmer would normally create issues, rather the testers should be doing that, however that option is available.

Go to Issue - Create issue, enter the issue attributes, and press Create. The issue is created in status Reported.

NOTE! An issue created in this way is not related to any test script and test step. This is suitable only for general issues that cannot be associated with any test script.

Resolve confirmed issues

Go to Issue - Confirmed issues screen, all the confirmed issues are displayed. Click on an issue Id, the Resolve issue screen is displayed.

If the issue is a bug in the tested system, fix the bug now.

Then select the Resolution as Fixed, select the Target system version in which the bug fix will be included and press the Resolve button. The issue Status becomes Resolved.

If the issue IS NOT a bug, resolve it as follows, select issue Resolution:

Enter the ResolverComment explaining why the issue has not been fixed and press Resolve. The issue status becomes Resolved.

NOTE! When resolving issue as either Invalid, Wontfix or Duplicate, the ResolverComment field must be filled.

NOTE! When resolving issue as Fixed, the Target field must be filled.

8 User guide - Support

8.1 Support request

Fulfill support requests

Go to Support - Pending support requests. Select a request you want to fulfill and click on its Id, the Support request detail screen is displayed with detailed information about the request.

Now perform the tasks that the tester is requesting.

Enter the information about performing the tasks and their results into the FulfillmentInfo field and press Done.

The FulfilledBy and FulfilledAt fields are filled in by the system and the request becomes Fulfilled.

Now go back to Pending support requests screen and continue with next request.

9 User guide - Common functionality

9.1 Test script

Print test script

Go to Test script - Print test script screen to save / print the current test script including the values filled in by the tester.

9.2 Issue

Issue discussion

On all the screens displaying the detail of an issue, there is an Issue discussion section on the bottom of the screen.

The issue discussion list displays the discussion posts in the order they were posted, together with their authors and time of posting.

The Issue discussion detail lets all users post new items into the discussion of the issue.

The discussion cannot be updated nor deleted.

NOTE! Use issue discussion to capture all information related to the issue for which there is no specific field on the issue itself.

9.3 Reports

Test run statistics

Go to Report - Test run statistics. The report displays list of all test runs together with statistical information on numbers of test scripts in the various statuses (Available, Open, Pass, Fail) and a total number of scripts.

Failed scripts

Go to Report - Failed scripts. The report displays list of all test scripts in status Fail. The list is not paged, so it can be easily copy-pasted into e.g. Excel for project reporting purposes.

Issue report

Go to Report - Issue report. The report displays list of history of issues. For each day on which an issue has been reported or changed its status in any way, the report includes a line. The line is marked with the date and includes the number of issues in the various statuses (Reported, Confirmed, Resolved, Verified) and a total number of issue to-date.

The report can be easily copy-pasted into Excel, and a nice graph can be created from it with a couple of mouse clicks (exactly 3 clicks, select the whole table excluding the Total column, and use the Stacked column graph).

The report provides an excellent overview of the testing and bug fixing effort of the team during the flow of the time.

An example of a real life graph presents the picture below.

Issue report graph

Issue overview

Go to Report - Issue overview. The report lists all the issues in a non-paged list, so the selected subset of the issues can be easily overlooked, or e.g. copy-pasted into any other application for project reporting purposes.

9.4 System

My user

Go to System - My user to change the attributes of your Mates user, including e.g. the login password.

10 Production notes

Mates has been used in production operation for about 18 months (as required by projects). It has been running on Red hat Linux & JBoss 3.2.3 and on Mandriva Linux 2006 & JBoss 3.2.7. A switch to JBoss 4.x.x is expected. Mates has always used the JBoss default Hypersonic database - without any problems. A journalled file system has been used on Mandriva (Reiser FS).

We shutdown JBoss and backup the database file daily (the big file found in <jboss_home>/server/default/data/hypersonic/ directory) and mail it to a public email address, but we have never needed the backups.

The application has been accessed on local LAN and via a VPN round the whole globe, both with a very nice response time.

Currently it is running on an old Pentium 630 server accessible from open internet. It is being used by some 5+ people every day with response time never complained about.