The test scripts are created in MS Excel and saved as XML. Then a provided tool is used by Mates test manager to merge a selected subset of tests into a single XML file, which is uploaded into Mates. Mates parses the file, extracts the included test scripts and creates corresponding records in its database.
The testers then login into Mates and fill in the test scripts during the testing. When finished, the testers mark the scripts as passed or failed.
The testers can also raise issues encountered during testing. The issues reported by testers are consolidated and confirmed by Mates test manager. All the confirmed issues are then resolved by the programmers. Finally, the resolved issues are subject to re-test by the test manager to verify that they were fixed.
In any moment, anyone can review the current, up-to-the-second status of the testing - numbers of test scripts that are passed, failed, or being worked on, number of issues that are reported, confirmed, or resolved, etc.
Mates also implements issue discussion, support request management, a standard, role-based user access system, and logging of user actions including scheduled purge of old log entries.
In addition, each of the three test runs described above can be also split into two, first including "smoke" tests, second including the detailed tests of the tested system. The smoke tests are simple, high-level overview tests that verify testability of the system. If any of the smoke tests fail, the testing is stopped - no detailed test scripts are executed until all the smoke tests pass.
In Mates, test scripts are organized in test runs. Every test script belongs to a specific test run. A test run is a wrapper for the test scripts. A test run must be created to upload test scripts into Mates. Additional test scripts can be subsequently uploaded into an open test run. A test run can be closed after all contained test scripts are closed (either passed or failed). There can be multiple test runs open at the same time.
The test manager can reopen a passed of failed script, and can also cancel assignment of a script from a tester.
Components primarily help differentiate the responsibility for fixing the bugs, so it is recommended that the components follow the separate development teams involved in the project. Alternatively, they can be defined on the level of the individual team members to track responsibility for fixing a particular bug to a single programmer.
NOTE! It is recommended that one of the components be "Test script". This way any potential problems in the test scripts themselves can be tracked in Mates and not forgotten to be fixed before the next test run.
Each fixed issue is assigned to another version of the system, documenting the version containing the fix.
The test manager defines the versions before the testing so that issues can be assigned to them.
The issues reported by testers are then reviewed by Mates test manager - he can re-word them, and change their severity and component assignment. Once updated, the test manager confirms the issues. Issues become Confirmed. At this stage, the test manager can also resolve duplicities (same issues reported by multiple testers), invalid issues, etc.
All the confirmed issues are then resolved by the programmers. Issues become Resolved.
Finally, the resolved issues are subject to re-test by the test manager to verify that they were fixed. Issues become Verified.
Test manager can re-confirm an issue from any status, i.e. also from Resolved (issue was not really resolved), and Verified (regression).
The tester creates the support request. Its status becomes Pending.
The support personnel perform the tasks requested, and mark the request as Fulfilled. The tester can then review the comments by the support personnel and continue with the execution of the test script.
| Role | Primary responsibilities |
| Test manager |
Maintain components.
Maintain versions. Create test runs. Upload test scripts into Mates. Close test runs. Confirm issues. Verify issues. |
| Tester |
Perform tests execution.
Report issues. Create support requests. |
| Programmer | Resolve issues. |
| Support | Fulfill support requests. |
| Administrator |
Maintain users.
Maintain user roles. Maintain system processes. |
| Review | [none] |
NOTE! Java JRE is not sufficient. JBoss requires JDK to compile JSP pages.
Install the JBoss application server properly - follow the JBoss documentation. Generally it is sufficient to set JAVA_HOME system property to the path where the Java JDK has been installed, unzip JBoss, and simply run it.
Extract the downloaded zip file into a chosen directory. The directory structure of the unzipped file is presented in the table below.
| Resource | Content |
| app | Development only. Contains the Mates definition XML file in SLIS (Small Language of Information Systems) - mates.xml. Contains all the externalized strings of Mates application - file matesLang.properties. |
| build | Contains the Java class files of the Mates tools used by the script processor tool. |
| dist | Contains the Mates distribution file - mates.ear. |
| doc | Contains the Mates documentation. |
| functional | Development only. Contains the automated Mates functional tests. The tests are automated using Canoo WebTest and DBUnit. |
| resources | Development only. Contains the HTML resources of Mates application. |
| script | Contains the script processor tool. |
| build.xml | Development only. This is the SEAF build file for the Mates application. |
| src.zip | Development only. Contains the Mates POJO model classes implementing its business logic in the SEAF framework. |
Copy the <mates_home>/dist/mates.ear file into the hot deploy directory of JBoss - <jboss_home>/server/default/deploy.
Optionally, if you are running JBoss in a console, watch deployment progress until it says that mates.ear is successfully deployed.
Now Mates is running under JBoss. Launch a web browser and point it to http://<hostname>:8080/mates/ (if your browser runs on the same machine as the JBoss server, use http://localhost:8080/mates/). When the page loads, you should see "Manual Tester home page" heading and a link "Enter the application..." underneath it.
Now the Mates installation has been successfully accomplished.
NOTE! This installation procedure assumes that JBoss has been installed as-provided and no additional configuration was done. In such a case, JBoss takes care of the deployment automatically: e.g. it creates the required DB tables in its default Hypersonic database, etc. If you deploy Mates on a JBoss installation with has been altered, please contact your JBoss administrator to help you with Mates deployment.
Define Domains in columns 10, 11. The domains should reflect functional areas in the tested system and thus categorize the test scripts. E.g. in Mates itself, the domains would be Test run, Script, Issue, etc.; in an ERP system the domains could be Product catalogue, Purchase Order, Delivery Note, Receiving Note, Invoice, etc.
Optionally modify Categories in column 13.
If you have changed the number of Domains (or Categories), select the Domain field B2 (or Category field B5), invoke menu Data - Validation, and update the Source accordingly.
Hide columns 7 to 13 again and save the template file.
Now fill-in the test script header. Choose Domain from the list (e.g. PO), enter the Code (e.g. 100) and Name (e.g. Create Purchase Order). Choose the Category (e.g. Core). Enter the Description and Prerequisites.
Now enter the steps of the test script. For each step enter the Task (e.g. Choose a supplier), Result name (e.g. Supplier number), and choose Result type from the list.
The result types supported by Mates are:
Now save the file. You have just written a Mates test script.
NOTE! The scripts are uniquely identified by a combination of the Domain and Code attributes. Any other attribute can change, but the script identity remains the same.
NOTE! Use Version to distinguish between different versions of the test script. Each time you modify a script, change its version. Keep all historical versions of the test scripts for reference.
NOTE! The blue colored fields at the right are included to support optional printing of the scripts and filling them on paper, as a fallback procedure. (Not that we had ever to use it...)
NOTE! Mates expects to receive the test scripts in UTF-8 character coding. Take care to avoid misinterpreting characters because of the encoding. From experience this can happen e.g. when MS Word is used to write the test script steps: (depending on setting) it replaces straight quotes with rounded quotes, which then cause problems when uploading to Mates, similarly replacing minus with long dash by MS Word causes the same problems.
NOTE! Review the PO100.xml and PO101.xml files in the <mates_home>/doc/example directory, they contain examples of created Mates test scripts. (Do not forget to open them in Excel.)
NOTE! The TestSuite.html overview can be produced by the script processor tool even without uploading test scripts to Mates. It is the best suited form e.g. for business people to review the test scripts before the start of the testing.
NOTE! Review the TestSuite.xml and TestSuite.html files in the <mates_home>/doc/example directory, they contain examples of generated Mates test suite files.
The User root detail screen is displayed. Enter "root" into the LoginName field and "password" into the Password field, and press "Create root" button.
The system displays confirmation message "The record has been created." Now follow the link "Log in..." in the top right corner of the browser to open the login screen, enter "root" into the LoginName field and "password" into the Password field, and press "Login" button.
Now the home screen should be displayed, and the "Logged in as: root" message should be indicated in the top right corner of the browser.
To review / modify role access rights (permissions), select the role on the Role search screen and go to screen Role permissions. For the selected role, a list of assigned permissions and a list of unassigned permissions are displayed. The assigned permissions list includes screen codes of screens accessible by a user with the role. The unassigned permissions list includes screen codes of screens which cannot be accessed by such user (with some exceptions, e.g. the login screen). To enable unassigned screen, click on its code - the code is moved from unassigned list to the assigned list. To disable assigned screen, click on its code - the code is moved from assigned list to the unassigned list.
You can recreate the user roles anytime in the future. This will restore the default roles with their default rights. However the users will have to be assigned to the roles after that again.
After you have created an Administrator account in the system, login again as Administrator, and perform all the following administration tasks under the Administrator account.
NOTE! Do not use the system from the root user. Create an administrator account to work with the system.
NOTE! Do not assign multiple roles to a single user unless actually required. It is recommended that each user has a single role. It is however possible to assign multiple roles to a single user.
If required, you can now start the Logger process: follow the Logger process name and press button Start. This begins creating log entries that can be reviewed at the System - Log screen.
If you intend to keep the logger on, it is recommended that you schedule log purging to prevent database overflow. On the System - Process search screen follow the PurgeLog process name. In the Process schedule list section select Hour 2 and Minute 00 and press Update. Then press EnableScheduling. This setting will purge all log records older than 1 month every day at 2 a.m.
NOTE! If using log, make sure the log purging is scheduled properly to avoid database overflow.
Now log in as a Test manager and continue with system setup in section 5.1.
The Mates system is now prepared to be routinely used. See the following section on how to load test scripts into the application.
Now the testers can log into Mates application and start filling the test scripts.
NOTE! If you have access to JBoss console or JBoss system.log file you can review the progress of processing the test scripts of the test suite file.
Go to Test run - Test run list and select the test run by clicking on its name. The Test run detail screen is displayed. Now go to Test run - Upload tests, copy the content of the TestSuite.xml file with the additional scripts into field TestSuite and press Upload.
Mates validates and processes the uploaded test scripts as follows:
NOTE! When the upload is aborted due to the fact that an open test script was included in the test suite file, remove the script from the <mates_home>/script/excel directory and re-generate the test suite file.
NOTE! A test run can only be closed if all its test scripts are closed (Pass or Fail). Finish all Open and Available test scripts before closing the test run.
Go to Script - All scripts, find the script (in status Pass or Fail) and click on its Name. Go to Maintain script and press button Reopen. The script status becomes Open.
Go to Script - All scripts, find the script (in status Open) and click on its Name. Go to Maintain script and press button CancelAssignment. The script status becomes Available.
Go to Issue - Create issue, enter the issue attributes, and press Create. The issue is created in status Reported.
NOTE! An issue created in this way is not related to any test script and test step. This is suitable only for general issues that cannot be associated with any test script.
Go to Issue - Issue search, find the issue and click on its Id. Go to Resolve issue. Choose issue Resolution:
NOTE! When resolving issue as either Invalid, Wontfix or Duplicate, the ResolverComment field must be filled.
NOTE! It is not expected that Test manager will resolve issues as Fixed. This is the job of the Programmers.
Now perform the retest on the tested system to verify that the issue has actually been resolved as indicated by the programmer.
If the re-test succeeds, enter the Verificaion Reference and press button Verify. The Verification Reference is e.g. a purchase order number on the test system which has been used for the verification re-test. The issue status becomes Verified. Go to Issue search screen again and continue with the next Resolved issue.
NOTE! If the retest fails, you have the option to re-confirm the issue again, if required - just press button Confirm and the issue status becomes Confirmed.
Go to Support - Support request search, find the request and click on its Id. Go to Support request detail screen, enter the FulfillmentInfo field and press Done. The request status becomes Fulfilled.
Press button Update. The script is set to status Open, and the TesterName and StartAt attributes are filled-in automatically by the system.
The script is from now not visible from the Available scripts screen, so that no other tester can take it. You, as its owner, can however see it in the My scripts screen.
Now you are on the Script detail screen and you can fill in the script steps, specifically the ResultValue column. There are 3 types of ResultTypes:
You can make the script passed by pressing the Pass button. The script must have all the steps filled in to make it passed. The script status becomes Pass and its EndAt attribute is filled in by the system.
You can make the script failed by pressing the Fail button at any time. The script status becomes Fail and its EndAt attribute is filled in by the system.
You can create an issue for a specific step by clicking on the Issue link next to the step. See Create issue section on how to fill in the values on the Create issue screen.
You can create a support request for a specific step by clicking on the Support link next to the step. See Create support request section on how to fill in the values on the Create support request screen.
Fill in the accurate and exact description of the problem into the Summary field. Select the Component and system Version in which the issue has been found. Select the issue Severity. Press the Create button.
The issue has now been created.
Now return to Script - Script detail screen and continue with filling in the script.
NOTE! An issue can also be created without a link on a script step, just by going to Issue - Create issue screen. However it is highly recommended to always create issue with a link to a particular script step to enable reproducibility of the issue by its future resolver.
Fill in the accurate and exact description of what you need to be done into the Todo field and press the Create button.
The support request has now been created.
You can now continue with another script, or wait until the support request gets fulfilled by a Support colleague by refreshing the Support - Support request search screen with the support request displayed until its status changes from Pending to Fulfilled.
When the support request becomes Fulfilled, click on its Id and on the Support request detail screen review the FulfillmentInfo field.
NOTE! A support request can also be created without a link on a script step, just by going to Support - Create support request screen. However it is highly recommended to always create support request with a link to a particular script step to provide a helpful context for the Support personnel when fulfilling the request.
Go to Issue - Create issue, enter the issue attributes, and press Create. The issue is created in status Reported.
NOTE! An issue created in this way is not related to any test script and test step. This is suitable only for general issues that cannot be associated with any test script.
If the issue is a bug in the tested system, fix the bug now.
Then select the Resolution as Fixed, select the Target system version in which the bug fix will be included and press the Resolve button. The issue Status becomes Resolved.
If the issue IS NOT a bug, resolve it as follows, select issue Resolution:
NOTE! When resolving issue as either Invalid, Wontfix or Duplicate, the ResolverComment field must be filled.
NOTE! When resolving issue as Fixed, the Target field must be filled.
Now perform the tasks that the tester is requesting.
Enter the information about performing the tasks and their results into the FulfillmentInfo field and press Done.
The FulfilledBy and FulfilledAt fields are filled in by the system and the request becomes Fulfilled.
Now go back to Pending support requests screen and continue with next request.
The issue discussion list displays the discussion posts in the order they were posted, together with their authors and time of posting.
The Issue discussion detail lets all users post new items into the discussion of the issue.
The discussion cannot be updated nor deleted.
NOTE! Use issue discussion to capture all information related to the issue for which there is no specific field on the issue itself.
The report can be easily copy-pasted into Excel, and a nice graph can be created from it with a couple of mouse clicks (exactly 3 clicks, select the whole table excluding the Total column, and use the Stacked column graph).
The report provides an excellent overview of the testing and bug fixing effort of the team during the flow of the time.
An example of a real life graph presents the picture below.
We shutdown JBoss and backup the database file daily (the big file found in <jboss_home>/server/default/data/hypersonic/ directory) and mail it to a public email address, but we have never needed the backups.
The application has been accessed on local LAN and via a VPN round the whole globe, both with a very nice response time.
Currently it is running on an old Pentium 630 server accessible from open internet. It is being used by some 5+ people every day with response time never complained about.