GUI testing tools serve the purpose of automating the testing process of software with graphical user interfaces.
Name | Supported platforms (testing system) | Supported platforms (tested system) | Developer | License | Automation | Latest version | Reference | Status |
---|---|---|---|---|---|---|---|---|
AscentialTest | Windows | Windows, Web | Zeenyx Software, Inc. | Proprietary | Yes | 9.0.1 | [1] | Active |
AutoIt | Windows | Windows | AutoIt | Proprietary | Yes | 3.3.14.0 | [2] | Active |
Appium | Windows, Linux, Mac (Python, C#, Ruby, Java, Javascript, PHP, Robot Framework) | iOS, Android (both native App & browser hosted app) | SauceLabs | Apache | Yes | (Binding Specific) | [3] | Active |
Dojo Objective Harness | cross-platform | Web | Dojo Foundation | AFL | Yes | 1.11.1 | [4] | Active |
eggPlant Functional | Windows, Linux, OS X | Windows, Linux, OS X, iOS, Android, Blackberry, Win Embedded, Win CE | TestPlant Ltd | Proprietary | Yes | Unknown | [citation needed] | Active |
HP WinRunner | Windows | Windows | Hewlett-Packard | Proprietary | Unknown | Unknown | [citation needed] | Discontinued |
IcuTest | Unknown | WPF | NXS-7 Software Inc | Proprietary | Unknown | Unknown | [citation needed] | Discontinued |
iMacros | Web (cross-browser) | Unknown | iOpus | Proprietary | Yes | 10.0.2 | [citation needed] | |
Katalon Studio | Windows, Linux, OS X | Web (UI & API), Mobile apps | Katalon LLC | Proprietary | Yes | 5.4.1 | [5] | Active |
Linux Desktop Testing Project | Linux (With Windows and OSX ports) | GUI applications with accessibility APIs | (Collaborative project) | GNU LGPL | Yes | 3.5.0 | [6] | |
Maveryx | Windows, Linux, OS X (only Java technologies) | Java, Swing, SWT, AWT, RCP, JavaFx, VB, MFC, .NET, WPF, HTML5 (cross-browser), | Maveryx Srl | Proprietary | Yes | 2.0.0 | [7] | Active |
Oracle Application Testing Suite | Windows | Web, Oracle Technology Products | Oracle | Proprietary | Yes | 12.5 | [8][9] | Active |
QF-Test | Windows, Linux, OS X, Web (cross-browser) | Java/Swing/SWT/Eclipse, JavaFX, Web applications | Quality First Software GmbH | Proprietary | Yes | 4.1.0 | [10] | Active |
Ranorex Studio | Windows | Windows, Web, iOS, Android | Ranorex GmbH | Proprietary | Yes | 8.3 | [11] | Active |
Rational Functional Tester | Windows, Linux | Windows, Swing, .NET, HTML | IBM Rational | Proprietary | Yes | 8.6.0.7 | [12] | Active |
Robot Framework | Web (cross-browser) | Web | (Collaborative project) | Apache | Yes | 3.0 | [13] | Active |
Sahi | Web (cross-browser), Windows | Web, Java, Java Web Start, Applet, Flex | Tyto Software[14] | Apache and Proprietary | Yes | 5.1 (open source), 7.0.1 | [15][16] | Active |
Selenium | Web (cross-browser) | Web | (Collaborative project) | Apache | Yes | 3.0 | [17] | Active |
SilkTest | Windows | Windows, Web | Micro Focus previously Borland and Segue | Proprietary | Yes | 17.5 | [18] | |
SOAtest | Windows, Linux, (cross-browser) | Web (cross-browser) | Parasoft | Proprietary | Yes | 9.10.3 | [19] | Active |
Squish GUI Tester | Windows, Linux, macOS, Solaris, AIX, QNX, WinCE, Windows Embedded, embedded Linux, Android, iOS | Qt, QML, QtQuick, Java AWT, Swing, SWT, RCP, JavaFx, Win32, MFC, WinForms, WPF, HTML5 (cross-browser), macOS Cocoa, iOS, Android, Tk | froglogic GmbH | Proprietary | Yes | 6.4 | [20] | Active |
Test Studio | Windows | Windows, Test Studio, Android, iOS | Telerik by Progress | Proprietary | Yes | 2017 R2 | [citation needed] | Active |
TestComplete | Windows | Windows, Android, iOS, Web | SmartBear Software | Proprietary | Yes | 12.20 | [citation needed] | Active |
Testing Anywhere | Unknown | Unknown | Automation Anywhere | Proprietary | Yes | 8.0 | [citation needed] | Discontinued |
TestPartner | Windows | Windows | Micro Focus | Proprietary | Yes | 6.3.1 | [citation needed] | Discontinued |
Tricentis Tosca | Windows | Windows, iOS, Android, Web, Cross-Browser, Java AWT, Java SWT, API, Win32, WinForms, WPF, Siebel, Delphi, PowerBuilder, up to around 40 different technologies | Tricentis | Proprietary | Yes | 12.0 | [21] | Active |
Twist | Unknown | Unknown | ThoughtWorks | Proprietary | Unknown | Unknown | [citation needed] | Discontinued |
Unified Functional Testing (UFT) previously named HP QuickTest Professional (QTP) | Windows | Windows, Web, Mobile, Terminal Emulators, SAP, Siebel, Java, .NET, Flex, others..[22] | Hewlett-Packard Enterprise | Proprietary | Yes | 14.00 | [23] | Active |
Visual Studio Coded UI Test | Windows | Windows, Web | Microsoft | Proprietary | Yes | 12.0 | Unknown | Inactive[24] |
Watir | Web | Web (cross-browser) | (Collaborative project) | BSD | Yes | 3.0 | [citation needed] | |
Xnee | UNIX | X Window | GNU Project, Henrik Sandklef | GNU GPL | Unknown | Unknown | [citation needed] |
References[edit]
- ^'Jolt Awards 2014: The Best Testing Tools'. Dr.Dobbs.com. June 3, 2014. Retrieved 2016-09-27.
- ^'AutoIt'. autolt. June 3, 2014. Retrieved 2016-09-27.
- ^'Mobile Phone automation'. appium. Retrieved 2018-12-11.
- ^'Dojo Toolkit'. DOJO. Retrieved 2016-09-27.
- ^Brian (2017-10-26). 'Best Automation Testing Tools for 2018 (Top 10 reviews)'. Medium. Retrieved 2018-05-22.
- ^'Linux Desktop Testing Project'. LDTP Website. Retrieved 2015-07-26.
- ^'Maveryx'. maveryx. Retrieved 2018-08-27.
- ^'Oracle - Oracle Application Testing Suite'. Retrieved 2016-11-08.
- ^'OATS - Oracle Application Testing Suite - Testingtools.co'.
- ^'QF-Test Overview'. QFS Website. Retrieved 2014-10-09.
- ^'Ranorex Release Notes'. Ranorex Website. Retrieved 2015-11-18.
- ^'Functional Tester'. IBM Website. Retrieved 2016-03-08.
- ^'GUI Automation'. Sorted Inf. Retrieved 2014-02-22.
- ^'Sahi Open Source Vs. Sahi Pro'. Sahi Pro Website. Retrieved 2014-09-02.
- ^'Sahi — Web Automation and Test Tool'. Open-source Sahi on Sourceforge. Retrieved 2014-09-02.
- ^'Sahi — The tester's Web Automation Tool'. Sahi Pro Website. Retrieved 2014-09-02.
- ^'What is Selenium?'. Selenium HQ. Retrieved 2014-09-02.
- ^'Silk Test Overview'(PDF).
- ^'Parasoft Launches SOATest: Raising the Bar on API Integrity'. 2013-06-14. Retrieved 2018-10-16.
- ^'Squish GUI Tester'. froglogic Website. Retrieved 2017-12-07.
- ^'Tricentis Tosca'. Tricentis Tosca. Retrieved 2017-04-12.
- ^'More Add-ins'. uft-help.saas.hpe.com. Retrieved 2017-08-02.
- ^'What's New in UFT 14.00'. uft-help.saas.hpe.com. Retrieved 2017-08-02.
- ^'Changes to Coded UI Test in Visual Studio 2019'. Azure DevOps Blog. Retrieved 2019-04-15.
I believe, and always try to explore other tools and techniques to find what more we can have to put in test automation. There are various feature that these tool carries with it and we should be aware of them, Wherever applicable we should try to use the other tools rather stick to the one or two specific tools.
However there are many great tools available, I have listed the commonly used five best test automation tools. If you know or have worked on any other tool than these please let us know. Share your experiences with these tools or any other via comments.
Test Automation Software Comparison
Telerik Test Studio
![Comparison Comparison](https://cdn-images-1.medium.com/max/1200/1*BI543mve6gSjON1Pp_pCKQ.png)
Telerik TestStudiois the easiest software testing tool you’ve tried – watch a couple of videos and you are ready to go. Navigate, point and click is all it takes to generate even the most complex of your functional, performance and load tests. It is an all in one testing software for functional, load, performance and mobile app testing. The in-depth functional testing includes native web and desktop apps testing along with mobile and tablet apps, HTML5, AJAX, Silverlight and WPF apps testing. Additionally testing teams can rely on the product to test JavaScript calls, Telerik controls, dynamic page synchronization, client-side behaviors, UI virtualizations and XAML animations. This software testing tool streamlines teamwork by introducing a common platform for testers and developers to work together. It ships with a standalone app and a Visual Studio plug-in that use the same repositories and file formats.
HP Unified Functional Testing Software
(Formerly HP QuickTest Professional)
A single solution for testing GUIs, APIs and multi layer applications
HP Unified Functional Testing (UFT) software is an automated software testing solution addressing the challenges of constant change in technology and processes. Automation testing is a leap forward in modern applications, and it can dramatically improve software quality while cutting testing costs and complexity even in the most rapidly changing environments. And with its integration with HP Application Lifecycle Management, it significantly enhances developer and tester productivity and collaboration.
Download Free Trial
Selenium
Selenium automates browsers. That’s it. What you do with that power is entirely up to you. Primarily it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.
Selenium has the support of some of the largest browser vendors who have taken (or are taking) steps to make Selenium a native part of their browser. It is also the core technology in countless other browser automation tools, APIs and frameworks.
TestComplete
TestComplete is a powerful and robust automated testing tool for mobile, web and desktop applications. Create accurate and repeatable automated tests across multiple devices, platforms, and environments quickly and easily – whether you are a complete beginner or an experienced automation engineer.
When testing today’s modern applications, it’s important that your tools are as agile and flexible as you need to be. TestComplete has the ability to script in multiple languages, supports modern control sets and integrates with popular open source frameworks and tools like Selenium, SoapUI and Jenkins. By giving you the ability to reuse tests across different devices, platforms and environments, TestComplete can significantly increase delivery speed, while helping save costs.
Under pressure to increase quality in faster release cycles? No problem. With TestComplete, you can create, manage and run tests for any mobile, web, or desktop software.
Free Automated Testing Tools
Know more about TestComplete
Watir
Watir, pronounced water, is an open-source (BSD) family of Ruby libraries for automating web browsers. It allows you to write tests that are easy to read and maintain. It is simple and flexible.
Watir drives browsers the same way people do. It clicks links, fills in forms, presses buttons. Watir also checks results, such as whether expected text appears on the page.
Watir will drive web applications that are served up as HTML pages in a web browser. Watir will not work with ActiveX plugin components, Java Applets, Macromedia Flash, or other plugin applications. To determine whether Watir can be used to automate a part of a web application, right click on the object and see if the View Source menu option is available. If you can view the HTML source, that object can be automated using Watir.
Like other programming languages, Ruby gives you the power to connect to databases, read data files and spreadsheets, export XML, and structure your code as reusable libraries. Unlike other programming languages, Ruby is concise and often a joy to read.
Hello Saket I am looking for the best automation tools to test forms which will be having barcodes on it and have to be decoded. It will be both paper and electronic forms mainly the tax forms.
When it comes to automated testing tools, there are a lot of automated testing tools currently on the market. And when you start searching for the right automated testing tool, it’s really hard for you to review all of the tools without comparison tables between these tools that you are confusing. Each of the tools has its own strengths and weaknesses, thus you have to have a deep understanding of their abilities to choose the right tool for automation testing.
Below are two comparison tables of four recommended automated testing tools (Katalon Studio, Selenium, UFT, TestComplete).
Table 1: Features supported.
Table 2: Strengths and weaknesses.
There is no one-size-fits-all tool for automated testing. But after reading this article, I hope you will have enough information to evaluate these tools and pick one that best meet your automated testing needs.
Table 1: Key features
Features | Katalon Studio | Selenium | UFT | TestComplete |
Test development platform | Cross-platform | Cross-platform | Windows | Windows |
Application under test | Web and mobile apps | Web apps | Windows desktop, Web, mobile apps | Windows desktop, Web, mobile apps |
Scripting languages | Java/Groovy | Java, C#, Perl, Python, JavaScript, Ruby, PHP | VBScript | JavaScript, Python, VBScript, JScript, Delphi, C++, and C# |
Programming skills | Not required. Recommended for advanced test scripts | Advanced skills needed to integrate various tools | Not required. Recommended for advanced test scripts | Not required. Recommended for advanced test scripts |
Learning curves | Medium | High | Medium | Medium |
Ease of installation and use | Easy to setup and run | Require installing and integrating various tools | Easy to setup and run | Easy to setup and run |
Script creation time | Quick | Slow | Quick | Quick |
Object storage and maintenance | Built-in object repository, XPath, object re-identification | XPath, UI Maps | Built-in object repository, smart object detection and correction | Built-in object repository, detecting common objects |
Image-based testing | Built-in support | Require installing additional libraries | Built-in support, image-based object recognition | Built-in support |
DevOps/ALM integrations | Many | No (require additional libraries) | Many | Many |
Continuous integrations | Popular CI tools (e.g. Jenkins, TeamCity) | Various CI tools (e.g. Jenkins, Cruise Control) | Various CI tools (e.g. Jenkins, HP Quality Center) | Various CI tools (e.g. Jenkins, HP Quality Center) |
Test Analytics | Katalon Analytics | No | No | No |
Product support | Ticketing support, community, dedicated staff (third-party support services) | Open source community | Dedicated staff, community | Dedicated staff, community |
License type | Freeware | Open source (Apache 2.0) | Proprietary | Proprietary |
Cost | Free | Free | License and maintenance fees | License and maintenance fees |
Table 2: Strengths and weaknesses
Tools | Strengths | Limitations |
Katalon Studio |
|
|
Selenium |
|
|
UFT |
|
|
TestComplete |
|
|
Source: Katalon Studio
In this article I will try to compare provided tools for SAP applications testing. This analisys were made for one of SAP customers.
There are three known options for SAP products testing:
- Using SAP Solution Manager Test Workbench
- Using HP tools (SAP Quality Center by HP)
- Using IBM Rational tools
At the same time, much attention is paid to testing automation which helps to solve such problems as:
- A large number of the tested business processes
- High labor input and cost of manual testing
- A human factor when testing
- Availability of necessary human resources for carrying out testing
- Complexity of scoping of testing
- Increase in complexity of testing in case of increase in number of business processes
However in the automated testing not everything is so simple.
High requirements of qualification of employees during creation of test scripts are imposed. It is also rather difficult to support the existing scripts in an actual condition in case of frequent change of business processes.
For the solution of similar problems there was developed so-called “Component-Based testing”
Component-Based testing of business processes allows to simplify creation and support of the automated test scripts for the users who are responsible for testing.
- Instead of working directly with a script, the tester can build the scenario, using ready components of the test, each of them represents the written-down script.
- The instrument of component testing scans the application and generates the automated script consisting of a set of components of the test.
- This script can be used further for testing. Testers can independently carry out such scanning since knowledge of programming of scripts isn’t required.
- The generated scenario can be united in an independent business component which can be used further for creation of other scripts
Time required for manual testing are the greatest for an overall picture of testing. However it is the most economic option for the scenarios tested once or very rare. Despite a possibility of reuse of the written-down automated scenarios, costs for their support are big since scripts improve manually. The smallest labor costs on support are required for component testing since there is a possibility of repeated scanning of the tested process without their rewriting.
The following evaluation criteria have been developed for further assessment, taking into account the aforesaid
Software Automated Testing Tools
Functional capabilities
Name of evaluation criterion | Description of evaluation criterion |
Test Plans generation | Possibility of drawing up plans of testing on the basis of the generated scripts. Possibility of use by the subsequent scripts of results previous. |
Existence of the tool for record of a test script | Existence of mechanisms for recording of the user actions made in system with a possibility to use this record at the subsequent testing. |
Existence of additional tools of component testing for optimization of works on generation and support of a test repository | Availability of additional instruments of record of the user actions facilitating initial generation of scripts with a possibility of their automated updating in case of change of the corresponding business process, for example SAP TAO |
Possibility of the automated testing of ABAP developments | Support of test automation of ABAP developments |
Possibility of the automated testing of Java developments | Support of automation of Java developments for the SAP applications |
Possibility of the automated testing of developments of SAP Webdynpro for Java | Support of automation of testing of developments of Webdynpro for Java |
Возможность автоматизированного тестирования разработок SAP Webdynpro for ABAP | Possibility of the automated testing of developments of SAP Webdynpro for ABAP |
Possibility of the automated testing of developments of SAP CRM Web UI | Support of automation of testing of developments of Web UI |
Analysis and Reporting | Possibility of generation of reports on testing and simplicity of their generation |
Support of regular load testing of SAP systems | Possibility of regular carrying out and comparison of several load tests on the chosen system in a uniform window with comparison of trends on the executed transactions |
Possibility of automation of testing of mobile platforms for SAP systems | Possibility of automation of functional regression testing of mobile applications on various devices |
Existence of a possibility of adjustment of loading at load testing | Adjustment of loading when conducting load testing |
Integration with application lifecycle management processes
Name of evaluation criterion | Description of evaluation criterion |
Integration with Project Documentation | Support of business processes structure. Possibility of transfer of structure of the project from SAP Solution Manager. Updating of structure of the SAP Solution Manager project based on results of accomplishment of scripts. Determination of a covering test scripts of specific elements of structure of business processes |
Integration with SAP Solution Manager ITSM scenarios | Possibility of maintaining errors of testing in Solution Manager. Existence of automatic communication of an incident and test unit |
Integration with Change Management of SAP SM | Possibility of use of the tool for testing of change operations of Solution Manager in a test landscape. Transfer of test results in change operation |
Integration with BPCA | Possibility to use of results of work of SAP Solution Manager Business Process Change Analyser (BPCA) intended for the analysis of changes in system with the subsequent generation and optimization of test plans |
Ease of use
Name of evaluation criterion | Description of evaluation criterion |
Simplicity of a script recording | Efforts on initial record of a script |
Simplicity of debugging of a script | Efforts for debugging of a script (adding of breakpoints, computing logic and etc) |
Simplicity of updating of a script | Efforts for updating of a script in case of change of business process in change transaction |
Simplicity of support of input data | Complexity of definition/updatingof input data of a script |
Usability | Simplicity of the interface and its clearness for the ultimate user |
Simplicity and convenience of administration | Simplicity and convenience of administration |
Existence of the built-in means of the organization of a role model | Existence of model of differentiations of powers in a product. |
On the basis of the criteria stated above, assessment of the offered tools has been carried out. Prioritezation of criteria has been executed by a customer on the basis of his requirements from 1 – not important, to 5 – very important. Opportunity/extent of realization can be expressed or as parameter realizable/unrealizable (1/0) or as extent of realization from 1 to 5
Functional capabilities analysis
Name of evaluation criterion | Priority | SAP SM | SAP QC by HP | IBM Rational | |
Test Plans generation | 4 | 1 | 1 | 1 | |
Existence of the tool for record of a test script | 4 | 1 | 1 | 1 | |
Existence of additional tools of component testing for optimization of works on generation and support of a test repository | 4 | 1 | 1 | 0 | |
Possibility of the automated testing of ABAP developments | 5 | 1 | 1 | 1 | |
Possibility of the automated testing of Java developments | 5 | 1 | 1 | 1 | |
Possibility of the automated testing of developments of SAP Webdynpro for Java | 5 | 1 | 1 | 1 | |
Возможность автоматизированного тестирования разработок SAP Webdynpro for ABAP | 5 | 1 | 1 | 1 | |
Possibility of the automated testing of developments of SAP CRM Web UI | 5 | 1 | 1 | 1 | |
Analysis and Reporting | 5 | 1 | 1 | 1 | |
Support of regular load testing of SAP systems | 5 | 0 | 1* | 1* | |
Possibility of automation of testing of mobile platforms for SAP systems | 5 | 1 | 1 | 0 | |
Existence of a possibility of adjustment of loading at load testing | 5 | 0 | 1 | 1 |
Integration with application lifecycle management processes analisys
Name of evaluation criterion | Priority | SAP SM | SAP QC by HP | IBM Rational |
Integration with Project Documentation | 4 | 1 | 1 | 1 |
Integration with SAP Solution Manager ITSM scenarios | 3 | 1 | 1 | 1 |
Integration with Change Management of SAP SM | 3 | 1 | 0 | 0 |
Integration with BPCA | 4 | 1 | 1 | 1 |
Ease of use analysis
Name of evaluation criterion | Priority | SAP SM | SAP QC by HP | IBM Rational |
Simplicity of a script recording | 5 | 5 | 5 | 5 |
Simplicity of debugging of a script | 4 | 5 | 5 | 3 |
Simplicity of updating of a script | 3 | 5 | 5 | 3 |
Simplicity of support of input data | 4 | 5 | 5 | 4 |
Usability | 5 | 5 | 5 | 5 |
Simplicity and convenience of administration | 3 | 5 | 5 | 5 |
Existence of the built-in means of the organization of a role model | 4 | 5 | 1 | 1 |
Tool cost | 5 | 5 | 3 | 4 |
Total:
• On the SAP Solution Manager 226 points
• On the HP 207 points
• On the IBM Rational 185 points
By results of assessment the SAP Solution Manager tools and HP tools are approximately identical by opportunities. Besides it is possible to carry to pluses of HP that this tool is long enough used in the market and is suitable for testing not only the SAP applications. But the SAP Solution Manager tools are free in the presence for Enterprise Support clients.
Further we will in detail consider the offered tools