Automation of Regression Testing in SAP S/4 HANA Environment

What are regression tests in the SAP environment?

Let me start with the main reasons for the dynamic changes in systems:

• growing user expectations,
• expanding the4 scale of business operations,
• development of technologies,
• new business environment.

All this means that technology and ERP systems are constantly evolving to meet the ever-increasing demands of organisations.

At Arvato Supply Chain Solutions, we rely primarily on the SAP system, which ensures the continuity of logistics operations in serving our customers. Such a system processes millions of orders and documents, so any optimisations and changes allow us to significantly increase the efficiency of operations and improve the comfort of work for hundreds of users.

It is worth noting, however, that the change process is not only an opportunity for the company but also a risk, which we minimise by means of appropriate system tests aimed at securing the entire process.

I would like to focus here on regression testing, the main role of which is to:

• confirm that the recently introduced changes to the program or code has not affected adversely other existing functions of the system or program,
• make sure that the changes have not resulted in new defects or previously undetected errors in the unchanged part of the software
• ascertain that the parts of the process that should not be affected by the change remain unaffected.

The abovementioned activities may prove to be more time consuming than just making the change. However, this is a cost that must be borne to minimise risk and ensure the highest quality of service.

The following diagram is used to illustrate the position and importance of regression tests:

The diagram shows that a change which is expected to influence the sorting process is an element that – as a rule – should trigger a whole chain of test activities. On the one hand, we have to make sure that the implemented change actually meets the business assumptions and works properly (sorting tests). On the other hand, however, in order to fully secure the process, it is necessary to perform additional tests of the entire process, possibly in many variants (regression tests).

Of course, the scope of activities and the need for testing depends on the specifics of the change and its nature, but the diagram is intended to provide an overview of the process.

Going through all the regression test scenarios is very time consuming and requires:

• Preparation of a large amount of test data (e.g. deliveries in SAP with appropriate parameters, available in many different process variants). Note that data preparation does not directly have any business value by itself. It is, however, an activity necessary to start the test itself.
• A test team will perform comprehensive regression tests within a defined time period. Depending on the specifics of the process, there may be tens or even hundreds of different test scenarios.

SAP regression test automation strategy

The above examples illustrate possible problems associated with complex regression tests. Understanding them is important in order to properly address the organisation’s effort required to perform them.

Before answering the question of whether the automated tests are worth it or not, let us try to illustrate what the automation actually is and what the target model of such a process could look like. The shipping process diagram referred to earlier will help us answer this question:

In this model, there is still a need to test the changes expected in the sorting process (sorting tests), but the entire regression testing activity is entrusted to a virtual machine (robot). An appropriately programmed robot will simulate the end user of the system, clicking on buttons in the application, entering values into appropriate fields and going through the entire business process in many variants in the SAP system.

The result of such actions (tests) can be returned at various levels and provide feedback to those involved in the change itself.

There are many reasons why the process of regression testing is worth automating and it would be difficult to name them all, but the main ones include:

• minimisation of human resources needed to perform regression tests,
• wider scope of tests, allowing a greater proportion of business requirements to be covered by tests,
• faster results and feedback for each test,
• possibility to engage staff in other types of tests (e.g. functional tests concerning processes that have changed),
• generation of test data for manual/functional tests (e.g. creation of many outgoing deliveries in many different configurations).

We should remember that earlier detection of defects and errors saves a lot of resources. The cost of locating and repairing a defect at a later stage of the system’s life cycle is much higher than if the defect is detected immediately after the change is made.

The example presented in the previous part of this article clearly illustrates the direction that our company is taking.

At Arvato SCS, a key element of testing is the test automation platform. Generally, there are a number of specialist tools available on the market for building an automated testing environment (including SAP), both for web and desktop applications.

In the model used by Arvato SCS, this role is fulfilled by the UiPath Test Suite tool, which, speaking in simple terms, offers:

• a user interface for the development of automated tests, providing the possibility of recording the process and performing logical operations, loops and interacting with all the elements of the SAP GUI application without the need for programming languages,
• a platform for users, allowing to run previously created automatic tests with various input parameters, which, if appropriately programmed, can influence the decisions or activities performed by the robot within the test,
• a possibility to connect multiple robots to the platform, on which automated tests will eventually be performed.

Virtual machines simulate a real operating system on which we can install various applications, including SAP.

Such a robot therefore has its own SAP GUI application with defined connections to SAP systems, which will be the object of regression test automation.

The robot is connected to the heart of the platform – the orchestrator – whose task is to start the pre-programmed test and transfer the task of its execution to the virtual machine mentioned before.

The conduct of the test is verified by a series of milestones programmed into the test, and the result itself is returned to the orchestrator in the form of a detailed log, screenshots and documents created in the SAP system.

In this way, the robot takes over the role of system tester, simulating the activities of the application user, thus allowing to take the burden of manual testing off the user. The testing team can focus on other types of system tests and leave repetitive regression testing tasks to the automation platform.

This model is the target direction the market is currently following, both in the SAP environment and in other web and desktop applications. Modern technology and automation of processes allows to eliminate errors and thus build competitive advantage of the organisation.

Therefore, the automation of regression testing is a trend that will allow the company in the long run not only to reduce operating costs and accelerate testing activities, but above all to ensure the security and stability of complex processes in the SAP environment.