Field Testing Strategy – Part #1
Guest Post by Dean Murphy:
Mobile applications have introduced a new paradigm for user interface design, technical architecture and support. When real-time communications is added the solution complexity increases significantly, suddenly there is an unreliable architectural component that neither the integrator nor the end client has any control over. The signal can vary and connections can stall, this variability and unreliability needs to be tested to see how the solution will perform and behave in the hands of the end-users.
Field operation of a mobile solution that incorporates real-time data exchange may exhibit symptoms that are not experienced in a laboratory testing environment.
- Application appears to hang in weak or no signal areas
- Application errors when connections timeout
- Runtime errors when unexpected conditions occur
- Stalled connections even in full signal areas
Regardless of the amount of testing carried out, the variable conditions of the field cannot be recreated in a lab setting. Testing in the lab alone could lead to problems when the solution is released to the field users.
An approach that has been successful has been devised that can stress test mobile solutions in the environment to which they will be deployed. This testing approach aims to recreate the conditions under which problems/issues may occur in the field. By building this approach into the test plans it will ensure that bugs and defects can be fixed prior to field trials and deployment.
The approach has been broken down into discreet sections:
The analyze section outlines the activities that should be conducted to facilitate the planning stage.
The table below suggests activities that should be carried out to identify resources and materials that will be used during field testing.
|Canvas the users||If field trials have taken place then talk to the field trial users to understand how the application behaves and try to identify any patterns to the issues they have observed.|
|Analyze usage patterns||Sometimes users will use the application in a way that suits their personal approach to work execution. This may differ from the documented business processes. Involve end-users to in test planning and use their experiences to build the test cases.|
|Identify work order volumes per user||Ascertain how many work orders a field operative completes per day and over what time period|
|Identify field test locations||Obtain cellular coverage maps from the mobile operator. Use these to identify test locations with strong, weak and no signal levels. See below for more information regarding the test location selection.|
|Identify test resources||The test resources must be familiar with the solution but should have a mindset that will question performance and stability and document their observations.|
|Drivers||Some test resources may not have driving licenses or the confidence to drive on unfamiliar roads. Consider assigning drivers to these individuals so that they can focus on testing the solution.|
|Test results analysis resource||A resource will be needed to analyze and report the test results, this person should also raise any bugs or defects identified during field testing.|
|Test vehicles||The use of private vehicles for field testing should be discouraged. Check the insurance policy to ascertain whether company vehicles can be used in this manner. If necessary, consider using hire vehicles with maximum insurance cover.|
|Identify peripherals||Ensure that all equipment and peripherals that will be the end users will use in relation to the solution have been identified (e.g. in-vehicle chargers, memory cards, cases, spare batteries, third-party software, etc.). Obtain and allocate this equipment for each test user.|
Test locations selection
The diagram below shows the location and coverage of cellular base stations around a town. Three test locations have been identified as accessible via public roads and were selected using the following criteria:
|Strong||Select an accessible area that is as close to a transmitter as possible. This will enable the application to perform at its best.|
|Weak||Choose a location that is between two transmitters and has marginal coverage from both. This will encourage the radio to hop between the transmitters and will test the solution’s ability to handle this.|
|None||Find an area that has no signal. These are difficult to find however if the device is placed in a transparent anti-static bag in a very weak signal area this will achieve the same effect and the application/device can still be accessed within the bag.|
Ensure that each location does not use the same transmitter and the test route should cross transmitter boundaries.
If the solution includes navigation software, enter the test locations as favorites for the convenience of the tester and to ensure that the same test location is used by all test resources.
Come back next week as we move on to the Plan phase…
About the Author:
Dean Murphy has been working in the field of mobility for 12 years and has broad experience of many mobile implementations from push email to large scale enterprise solutions. Dean is a Solution Consultant with ClickSoftware and helps customers understand how they can solve their business problems through the use of the ClickSoftware Mobility Suite and the ServiceOptimization Suite.