General Test Procedures
There are two major test procedures for functional test at network operators:
- End to end tests, where an user equipment (UE, eg. a computer or a phone) is operated
- Simulation tests, where the signaling is simulated to pretend the existence of users and devices, using the service
Other tests are handling load or smoke tests as well as security tests. Both of these tests are normally handled by systems, not really involved in functionality.
Last but not least, operators often use heuristic methods to detect deviations in e.g. log files or performance graphs to accept them as a symptom of a still to be found error. This is interesting, but extremely expensive to develop and to maintain. In addition, these procedures grow over years and personnel fluctuation makes know-how transfer more difficult.
This is also not really meant to be a ‘test procedure’, but we like to mention it in the context of test automation. These methods at least provide some form of automated scepticism, which could be used to trigger the test automation system.
Degust Test Procedures
Due to the variety of systems at the operator’s end, the industry needs a flexible solution that combines these test procedures and incorporates results, provided by 3rd party systems.
On this page, we will differentiate the various test procedures and how Degust eliminates any disadvantages.
Because you are reading this in context of our Degust test automation system, we won’t highlight the advantages of test automation over manual tests.
Automated Real End to End Testing
- All tests show the view of the end customer
- the complete E2E chain is tested
- a registered SIM is required
- Tests can take a long time (session setup, download times etc.).
- Negative tests are impossible (e.g. faulty communication PGW/AAA)
- Parallelisation is only possible with high device and SIM costs
- Troubleshooting is difficult. Evaluation only from the user’s point of view, no view of the systems,
- Log file evaluation in live environment not always possible/sensible (security)
- Devices are required in all locations to be tested
Simulation of Core Interface Signaling
- All scenarios can be mapped “on the cable” to the services
- Negative tests are possible e.g. inserting wrong attributes into the packets
- System responses can be evaluated in any depth, because the test machine is the client and thus receives the responses
- SIM cards do not have to be available/registered (although it would make testing easier)
- Test times are greatly reduced, e.g. because downloads do not have to be waited for
- Because on end user device is involved, almost unlimited parallelism is possible
- Significantly higher test density
- Tests also possible with locations at which no E2E devices exist
- Tests are also possible before the installation of real devices and services (e.g. POC, type acceptances etc.)
- The tests do not show the end-user’s view (no screenshots, etc.)
Interfacing to 3rd Party Systems
- Provisioning of test conditions subscribers, tariff plans, devices, etc.
- Start/stop logging and traces collect evidence
- Starting/stopping processes operation automation
- download of evidence files, screenshots …
- API queries
- Controlling ticket flows in troubleshooting systems
- Reporting and triggering to O&M systems
- Some OEMs might raise concerns
- Sensitive planning is necessary in production environments
Degust combines the best procedures
Degust® completely eliminates the con-arguments of the 3 processes and offers the operator the necessary flexibility to run the optimal tests in each production step and to evaluate the results completely automatically.
If, for example, the tests used during type approval phases are also used during the system’s live introduction, a pure E2E test system would limit the test density without need you always have to wait for free devices or during slow downloads.
Degust combines the E2E tests with the simulation, e.g. to consume additional volume during necessary E2E tests or to deduct subscriber’s prepaid account to test the effects on the online charging systems.
Remember: Speed and depth of testing are essential during product launch.