Mates manual

1.1 Overview

The test scripts are created in MS Excel and saved as XML. Then a provided tool is used by Mates test manager to merge a selected subset of tests into a single XML file, which is uploaded into Mates. Mates parses the file, extracts the included test scripts and creates corresponding records in its database.

The testers then login into Mates and fill in the test scripts during the testing. When finished, the testers mark the scripts as passed or failed.

The testers can also raise issues encountered during testing. The issues reported by testers are consolidated and confirmed by Mates test manager. All the confirmed issues are then resolved by the programmers. Finally, the resolved issues are subject to re-test by the test manager to verify that they were fixed.

In any moment, anyone can review the current, up-to-the-second status of the testing - numbers of test scripts that are passed, failed, or being worked on, number of issues that are reported, confirmed, or resolved, etc.

Mates also implements issue discussion, support request management, a standard, role-based user access system, and logging of user actions including scheduled purge of old log entries.

1.2 Test run

In addition, each of the three test runs described above can be also split into two, first including "smoke" tests, second including the detailed tests of the tested system. The smoke tests are simple, high-level overview tests that verify testability of the system. If any of the smoke tests fail, the testing is stopped - no detailed test scripts are executed until all the smoke tests pass.

In Mates, test scripts are organized in test runs. Every test script belongs to a specific test run. A test run is a wrapper for the test scripts. A test run must be created to upload test scripts into Mates. Additional test scripts can be subsequently uploaded into an open test run. A test run can be closed after all contained test scripts are closed (either passed or failed). There can be multiple test runs open at the same time.

1.3 Test script

The test manager can reopen a passed of failed script, and can also cancel assignment of a script from a tester.

1.4 Component

Components primarily help differentiate the responsibility for fixing the bugs, so it is recommended that the components follow the separate development teams involved in the project. Alternatively, they can be defined on the level of the individual team members to track responsibility for fixing a particular bug to a single programmer.

NOTE! It is recommended that one of the components be "Test script". This way any potential problems in the test scripts themselves can be tracked in Mates and not forgotten to be fixed before the next test run.

1.5 Version

Each fixed issue is assigned to another version of the system, documenting the version containing the fix.

The test manager defines the versions before the testing so that issues can be assigned to them.

1.6 Issue

The issues reported by testers are then reviewed by Mates test manager - he can re-word them, and change their severity and component assignment. Once updated, the test manager confirms the issues. Issues become Confirmed. At this stage, the test manager can also resolve duplicities (same issues reported by multiple testers), invalid issues, etc.

All the confirmed issues are then resolved by the programmers. Issues become Resolved.

Finally, the resolved issues are subject to re-test by the test manager to verify that they were fixed. Issues become Verified.

Test manager can re-confirm an issue from any status, i.e. also from Resolved (issue was not really resolved), and Verified (regression).

1.7 Support request

The tester creates the support request. Its status becomes Pending.

The support personnel perform the tasks requested, and mark the request as Fulfilled. The tester can then review the comments by the support personnel and continue with the execution of the test script.

1.8 Data model

1.9 User roles overview

2.1 Java JDK

3.1 Configuring the script template

Define Domains in columns 10, 11. The domains should reflect functional areas in the tested system and thus categorize the test scripts. E.g. in Mates itself, the domains would be Test run, Script, Issue, etc.; in an ERP system the domains could be Product catalogue, Purchase Order, Delivery Note, Receiving Note, Invoice, etc.

Optionally modify Categories in column 13.

If you have changed the number of Domains (or Categories), select the Domain field B2 (or Category field B5), invoke menu Data - Validation, and update the Source accordingly.

Hide columns 7 to 13 again and save the template file.

3.2 Writing a new script

Now fill-in the test script header. Choose Domain from the list (e.g. PO), enter the Code (e.g. 100) and Name (e.g. Create Purchase Order). Choose the Category (e.g. Core). Enter the Description and Prerequisites.

Now enter the steps of the test script. For each step enter the Task (e.g. Choose a supplier), Result name (e.g. Supplier number), and choose Result type from the list.

The result types supported by Mates are:

• b - Boolean, can be Yes or No.
• n - Number, no alphabetical characters are allowed.
• t - Text, any characters allowed.

Now save the file. You have just written a Mates test script.

NOTE! The scripts are uniquely identified by a combination of the Domain and Code attributes. Any other attribute can change, but the script identity remains the same.

NOTE! Use Version to distinguish between different versions of the test script. Each time you modify a script, change its version. Keep all historical versions of the test scripts for reference.

NOTE! The blue colored fields at the right are included to support optional printing of the scripts and filling them on paper, as a fallback procedure. (Not that we had ever to use it...)

NOTE! Mates expects to receive the test scripts in UTF-8 character coding. Take care to avoid misinterpreting characters because of the encoding. From experience this can happen e.g. when MS Word is used to write the test script steps: (depending on setting) it replaces straight quotes with rounded quotes, which then cause problems when uploading to Mates, similarly replacing minus with long dash by MS Word causes the same problems.

NOTE! Review the PO100.xml and PO101.xml files in the <mates_home>/doc/example directory, they contain examples of created Mates test scripts. (Do not forget to open them in Excel.)

3.3 Creating a test suite

• TestSuite.xml is the file prepared to be uploaded into Mates application.
• TestSuite.html is a HTML overview of all the test scripts included in the test suite. It serves as a handy quick reference, where all the test scripts can be reviewed (searched, etc.) at once.

NOTE! The TestSuite.html overview can be produced by the script processor tool even without uploading test scripts to Mates. It is the best suited form e.g. for business people to review the test scripts before the start of the testing.

NOTE! Review the TestSuite.xml and TestSuite.html files in the <mates_home>/doc/example directory, they contain examples of generated Mates test suite files.

4.1 First use - system setup before testing

Create the root user

The User root detail screen is displayed. Enter "root" into the LoginName field and "password" into the Password field, and press "Create root" button.

The system displays confirmation message "The record has been created." Now follow the link "Log in..." in the top right corner of the browser to open the login screen, enter "root" into the LoginName field and "password" into the Password field, and press "Login" button.

Now the home screen should be displayed, and the "Logged in as: root" message should be indicated in the top right corner of the browser.

Create user roles

To review / modify role access rights (permissions), select the role on the Role search screen and go to screen Role permissions. For the selected role, a list of assigned permissions and a list of unassigned permissions are displayed. The assigned permissions list includes screen codes of screens accessible by a user with the role. The unassigned permissions list includes screen codes of screens which cannot be accessed by such user (with some exceptions, e.g. the login screen). To enable unassigned screen, click on its code - the code is moved from unassigned list to the assigned list. To disable assigned screen, click on its code - the code is moved from assigned list to the unassigned list.

You can recreate the user roles anytime in the future. This will restore the default roles with their default rights. However the users will have to be assigned to the roles after that again.

Create users

After you have created an Administrator account in the system, login again as Administrator, and perform all the following administration tasks under the Administrator account.

NOTE! Do not use the system from the root user. Create an administrator account to work with the system.

NOTE! Do not assign multiple roles to a single user unless actually required. It is recommended that each user has a single role. It is however possible to assign multiple roles to a single user.

Create processes

If required, you can now start the Logger process: follow the Logger process name and press button Start. This begins creating log entries that can be reviewed at the System - Log screen.

If you intend to keep the logger on, it is recommended that you schedule log purging to prevent database overflow. On the System - Process search screen follow the PurgeLog process name. In the Process schedule list section select Hour 2 and Minute 00 and press Update. Then press EnableScheduling. This setting will purge all log records older than 1 month every day at 2 a.m.

NOTE! If using log, make sure the log purging is scheduled properly to avoid database overflow.

Now log in as a Test manager and continue with system setup in section 5.1.

5.1 System setup before testing

Components

Versions

The Mates system is now prepared to be routinely used. See the following section on how to load test scripts into the application.

5.2 Test run

Create test run

Now the testers can log into Mates application and start filling the test scripts.

NOTE! If you have access to JBoss console or JBoss system.log file you can review the progress of processing the test scripts of the test suite file.

Uploading additional test scripts

Go to Test run - Test run list and select the test run by clicking on its name. The Test run detail screen is displayed. Now go to Test run - Upload tests, copy the content of the TestSuite.xml file with the additional scripts into field TestSuite and press Upload.

Mates validates and processes the uploaded test scripts as follows:

• If the parsed script does not exist in the current open test run yet (matching by combination of Domain and Code), it is accepted and CREATED.
• If the parsed script exists in the current test run already and is in status Available (the script was not used by any tester yet), it is accepted and REPLACED (the script header is updated, the old steps are deleted and created again).
• If the parsed script exists in the current test run already and is in status other than Available (the script was already used by a tester), it is not accepted and an ERROR message is displayed. The upload of the scripts is aborted.

NOTE! When the upload is aborted due to the fact that an open test script was included in the test suite file, remove the script from the <mates_home>/script/excel directory and re-generate the test suite file.

Close test run

NOTE! A test run can only be closed if all its test scripts are closed (Pass or Fail). Finish all Open and Available test scripts before closing the test run.

5.3 Test script

Reopen test script

Go to Script - All scripts, find the script (in status Pass or Fail) and click on its Name. Go to Maintain script and press button Reopen. The script status becomes Open.

Cancel assignment of a test script

Go to Script - All scripts, find the script (in status Open) and click on its Name. Go to Maintain script and press button CancelAssignment. The script status becomes Available.

5.4 Issue

Create issue

Go to Issue - Create issue, enter the issue attributes, and press Create. The issue is created in status Reported.

NOTE! An issue created in this way is not related to any test script and test step. This is suitable only for general issues that cannot be associated with any test script.

Confirm reported issues

Eliminate wrong issues

Go to Issue - Issue search, find the issue and click on its Id. Go to Resolve issue. Choose issue Resolution:

• Invalid issue means the reporter made a mistake when reporting it, or e.g. the issue cannot be reproduced.
• Wontfix means it is an issue alright, but a decision has been made not to fix it. This can happen e.g. when for an issue in system A is rather made a work-around in system B (which is e.g. processed as another issue with severity Enhancement).
• Duplicate means the issue is correct, but has already been reported and processed under another Id. Note the Id of the primary issue in the ResolverComment field.

NOTE! When resolving issue as either Invalid, Wontfix or Duplicate, the ResolverComment field must be filled.

NOTE! It is not expected that Test manager will resolve issues as Fixed. This is the job of the Programmers.

Verify resolved issues

Now perform the retest on the tested system to verify that the issue has actually been resolved as indicated by the programmer.

If the re-test succeeds, enter the Verificaion Reference and press button Verify. The Verification Reference is e.g. a purchase order number on the test system which has been used for the verification re-test. The issue status becomes Verified. Go to Issue search screen again and continue with the next Resolved issue.

NOTE! If the retest fails, you have the option to re-confirm the issue again, if required - just press button Confirm and the issue status becomes Confirmed.

5.5 Support request

Fulfill pending support request

Go to Support - Support request search, find the request and click on its Id. Go to Support request detail screen, enter the FulfillmentInfo field and press Done. The request status becomes Fulfilled.

6.1 Test script

Take an available script

Press button Update. The script is set to status Open, and the TesterName and StartAt attributes are filled-in automatically by the system.

The script is from now not visible from the Available scripts screen, so that no other tester can take it. You, as its owner, can however see it in the My scripts screen.

Fill in script

Now you are on the Script detail screen and you can fill in the script steps, specifically the ResultValue column. There are 3 types of ResultTypes:

• Boolean - accepts only Yes or No. Yes can be typed lowercase or uppercase, shortened as y, or alternatively entered as true (t). Similarly No can be also shortened as n, or entered as false (f). Mates automatically expands these to Yes or No.
• Number - accepts numerical characters and punctuation. Only alphabetic characters are not accepted. This allows not just to enter a number (even decimal), but also e.g. a list of numbers.
• Text - accepts any characters. There is one exception: Text does not accept anything that is interpreted by Boolean result as Yes, so e.g. you cannot enter plain y into a Text result. This is to prevent the testers to mindlessly enter y everywhere.

You can make the script passed by pressing the Pass button. The script must have all the steps filled in to make it passed. The script status becomes Pass and its EndAt attribute is filled in by the system.

You can make the script failed by pressing the Fail button at any time. The script status becomes Fail and its EndAt attribute is filled in by the system.

You can create an issue for a specific step by clicking on the Issue link next to the step. See Create issue section on how to fill in the values on the Create issue screen.

You can create a support request for a specific step by clicking on the Support link next to the step. See Create support request section on how to fill in the values on the Create support request screen.

6.2 Issue

Create issue

Fill in the accurate and exact description of the problem into the Summary field. Select the Component and system Version in which the issue has been found. Select the issue Severity. Press the Create button.

The issue has now been created.

Now return to Script - Script detail screen and continue with filling in the script.

NOTE! An issue can also be created without a link on a script step, just by going to Issue - Create issue screen. However it is highly recommended to always create issue with a link to a particular script step to enable reproducibility of the issue by its future resolver.

6.3 Support request

Create support request

Fill in the accurate and exact description of what you need to be done into the Todo field and press the Create button.

The support request has now been created.

You can now continue with another script, or wait until the support request gets fulfilled by a Support colleague by refreshing the Support - Support request search screen with the support request displayed until its status changes from Pending to Fulfilled.

When the support request becomes Fulfilled, click on its Id and on the Support request detail screen review the FulfillmentInfo field.

NOTE! A support request can also be created without a link on a script step, just by going to Support - Create support request screen. However it is highly recommended to always create support request with a link to a particular script step to provide a helpful context for the Support personnel when fulfilling the request.

7.1 Issue

Create issue

Go to Issue - Create issue, enter the issue attributes, and press Create. The issue is created in status Reported.

NOTE! An issue created in this way is not related to any test script and test step. This is suitable only for general issues that cannot be associated with any test script.

Resolve confirmed issues

If the issue is a bug in the tested system, fix the bug now.

Then select the Resolution as Fixed, select the Target system version in which the bug fix will be included and press the Resolve button. The issue Status becomes Resolved.

If the issue IS NOT a bug, resolve it as follows, select issue Resolution:

• Invalid issue means the reporter made a mistake when reporting it, or e.g. the issue cannot be reproduced.
• Wontfix means it is an issue alright, but a decision has been made not to fix it. This can happen e.g. when for an issue in system A is rather made a work-around in system B (which is e.g. processed as another issue with severity Enhancement).
• Duplicate means the issue is correct, but has already been reported and processed under another Id. Note the Id of the primary issue in the ResolverComment field.

NOTE! When resolving issue as either Invalid, Wontfix or Duplicate, the ResolverComment field must be filled.

NOTE! When resolving issue as Fixed, the Target field must be filled.

8.1 Support request

Fulfill support requests

Now perform the tasks that the tester is requesting.

Enter the information about performing the tasks and their results into the FulfillmentInfo field and press Done.

The FulfilledBy and FulfilledAt fields are filled in by the system and the request becomes Fulfilled.

Now go back to Pending support requests screen and continue with next request.

9.1 Test script

Print test script

9.2 Issue

Issue discussion

The issue discussion list displays the discussion posts in the order they were posted, together with their authors and time of posting.

The Issue discussion detail lets all users post new items into the discussion of the issue.

The discussion cannot be updated nor deleted.

NOTE! Use issue discussion to capture all information related to the issue for which there is no specific field on the issue itself.

9.3 Reports

Test run statistics

Failed scripts

Issue report

The report can be easily copy-pasted into Excel, and a nice graph can be created from it with a couple of mouse clicks (exactly 3 clicks, select the whole table excluding the Total column, and use the Stacked column graph).

The report provides an excellent overview of the testing and bug fixing effort of the team during the flow of the time.

An example of a real life graph presents the picture below.

Issue overview

9.4 System

My user

We shutdown JBoss and backup the database file daily (the big file found in <jboss_home>/server/default/data/hypersonic/ directory) and mail it to a public email address, but we have never needed the backups.

The application has been accessed on local LAN and via a VPN round the whole globe, both with a very nice response time.

Currently it is running on an old Pentium 630 server accessible from open internet. It is being used by some 5+ people every day with response time never complained about.