Automated Testing for Mainframe Modernization

Automated testing for mainframe modernization. How does it work?

Hi, I’m Geert, product manager for the Anubex testing tools. I’m going to explain how we use automated testing for our mainframe modernizations.

You can learn about the concepts of automated testing in the context of mainframe modernization in another video. Here I want to show how these concepts are applied in practice.

Before we start, it’s important to explain that our migration projects are extremely iterative processes. Roughly from day one until the go-live we perform the same steps: we 1) convert the entire application, 2) run the entire test set, 3) analyze differences between desired and actual behavior, 4) correct some issues we encounter. And then we repeat.

Because we use tools to do the bulk of the work, these iterations can happen very quickly, say a few hours to convert and test an application of a few million lines. This allows us to focus on correcting issues. We’ll get near instant feedback on whether the issue is indeed fixed, and whether any other new issues were introduced.

Apart from this let’s call it “daily cycle” of convert/test/tune, there is also a bigger cycle that repeats roughly every three to four months. At the start of each such “big cycle" we take a snapshot of the application at that moment in time. During the next few months the migration project will operate on that static copy, while the real application continues to evolve in parallel. As we enter the next project phase, we again synchronize the migration project with the then current production state of the application.

This allows us to have a relatively stable migration environment, while daily maintenance on the production environment can continue unhindered, and in parallel to the migration project. This means we have zero code freeze during the entirety of the migration project!

Like I said, at the start of each big cycle, a snapshot of the application is taken. That snapshot contains all application source code and configuration. Obviously, because that’s what we are going to migrate. It also contains some test data and test scenarios that the customer will provide us. These tests will serve as the acceptance criteria for that phase. During the remainder of that phase, we are going to run the daily cycles on this snapshot: convert the code and the data, run the tests, see if it works, and fix where needed. If all tests succeed, we know we have achieved the goals for that phase and can advance to the next major cycle.

So now let’s zoom in on the processes behind the tests. Here too, everything is automated to the max. There are three distinct actions: 1) recording 2) playback 3) analysis

Recording happens on the reference environment, the original application. We offer several ways to record test scenarios, but the easiest in my opinion is the so-called Proxy Recording. TestMatch sits between the terminal and the mainframe. The testers simply use the application to perform some actions. Their interactions are captured in TestMatch while they pass to and from the mainframe, and they instantly become the test scenario.

Once the scenario has been recorded, we can replay it over and over again onto the migrated application. No human interaction is needed anymore. User input that was previously recorded is now submitted automatically to the migrated application. The application responses are again recorded for subsequent analysis. During replay we can optionally suppress user think time from the original recording, to make the replay even faster.

The third step finally is to compare the original recording with the replay recording. The main premise of our “like for like migration” is that the original and the migrated application should behave completely identical. So, given the same inputs, the migrated application should produce the same outputs. The test tool will compare each screen for differences: in text content, in attributes like text color, even invisible text is compared. Differences are highlighted and can be inspected in their context. Expected differences like time stamps, dates… can be marked “to be ignored”. Real differences can be submitted for further analysis and follow-up in a bug tracking system, directly from within TestMatch.

We can compare functional behavior, like screen content, but also non-functional behavior, like performance. Our test tools can compare original response time with the response time of the migrated application, to ensure that performance is at least as good as it was on the mainframe. This is one of the three promises we make for each of our migration projects: functional equivalence, performance equivalence and maintainability equivalence.

Our automated testing tools save hundreds of man-days in typical migration projects, by allowing the tests to be replayed continuously without any human interaction. Our migration specialists can focus on locating and solving issues in the migration.

But the tools also provide some very clear metrics to the project board, about the state and the evolution of the project. The number of daily tests succeeded and failed, put in Excel, provide a great summary of past performance, and trendlines for the future.

So, to conclude, I think it’s fair to say that our test tools are a vital part of any migration project. It is reassuring to know that every aspect of the project is being constantly tested. In fact, this is key in building up confidence with all stakeholders that by the time of go-live, the migrated application will have the stability and maturity it requires to move into production.

If you have any questions or remarks, feel free to leave a comment below, or visit anubex.com for more resources.

Thanks for watching!

‍