DataTurn: An All-in-One Database Conversion

In March, Astadia announced the availability of the FastTrack Factory™, which automates the process of moving IBM and Unisys mainframe applications and databases to distributed and cloud computing platforms. In a previous post, we provided an overview of the platform, including its four key components, CodeTurn, DataTurn, TestMatch and DataMatch.  

In this article, we’ll drill down into further detail on DataTurn specifically, including that tool’s capabilities for mapping and converting legacy databases including Adabas, IDMS, DB2, IMS, VSAM, ISAM, and more. DataTurn converts data from legacy systems to virtually any modern relational database platform, working in conjunction with the other three tools that make up the backbone of the Migration Factory. Just as importantly, it provides powerful data conversion capabilities with full automation.

Industrialized Data Migration

Let’s briefly review the Migration Factory’s four main building blocks: CodeTurn automates the refactoring of legacy code into modern programming languages with automation levels of approximately 99.99%. DataTurn automates the conversion of data from legacy databases such as IDMS and Natural ADABAS to virtually any modern relational database, – including SQL Server, Oracle, IBM DB2, PostgreSQL, and others.

TestMatch is a test automation tool that uses tracing capabilities within source IBM or Unisys platforms to record user activity in the live production system, and then to replicate that user activity in the target system. Finally, DataMatch performs a comparison of source and target databases, – automatically adjusting for any modifications that might have been made to the database schema as part of the migration process.

The four main components of the Migration Factory work together so that refactoring and testing are automated and harmonized using fully automated processes.  That industrialized migration process allows for virtually unlimited testing before a clean cutover from legacy systems to the new platform.  This vastly reduces risk while cutting the duration of a typical migration project by as much as 90%.

DataTurn is a critical piece of the overall migration picture, not merely because it maps data from the old database to a newer one; it also allows for conversion and normalization of legacy data into a relational structure in the target system. In other words, DataTurn enables customers to modify data structures to better suit their needs following the data conversion.

Most legacy databases, for example, store data in ways that defy the fundamental principles of relational databases. Redefines, for example, are common in many legacy systems, – in effect storing more than one value in a single field and parsing those values at runtime. Occurrences, repeating groups, and other non-relational concepts have been implemented in virtually all legacy databases.

DataTurn enables users to re-map these kinds of anomalies to the target database, where they can be redesigned and stored according to relational principles.  For example, in a legacy application that allows each customer record to be associated with 10 different phone numbers, the original schema might have been designed as a flat record, with each customer record containing 10 separate phone number fields (Phone_01, Phone_02, etc.). In a relational database, it would generally be preferable to store customers’ phone number information in a separate table that links on the customer ID. In this way, database designers can transition from a flat (and less flexible) data structure to a relational one-to-many model.

This is where the Migration Factory is such a powerful tool: when data structures differ between source and target databases, application code would normally need to be modified to accommodate the differences. DataTurn works together with CodeTurn to ensure that refactored code is generated in such a way that it can continue to access the right data, even when the underlying data structures have changed.

Using the phone number example cited above, a direct mapping of the legacy data structure might result in a query that looks something like this:

SELECT CustName, Phone_02 from Customers

WHERE Customer_ID = “ABC123”

In a relational context, this would look a bit different:

SELECT Customers.CustName, CustPhone.PhoneNumber from Customers, CustPhone

WHERE Customers.Customer_ID = “ABC123” and CustPhone.Phone_ID=2

INNER JOIN Customers ON Customers.Customer_ID = CustPhone.Customer_ID

DataTurn automatically accommodates this change by creating an abstraction layer for data access, which we call the I/O Module.  In other words, as the code is re-factored, database queries are moved into the abstraction layer, where mapping between the old database structure and the new one is handled automatically. The re-factored code does not make direct database calls; rather it speaks to the I/O Module, which serves as a kind of on-the-fly translator. This gives us full automation and complete control of the process.

Clearly there are some advantages in moving to a relational structure; however, we should point out that performance can be just a bit slower for relational queries.  That’s why it’s so important to provide this kind of flexibility in mapping the legacy database structure to the target environment.  DataTurn works seamlessly with the other components of the Migration Factory to make this possible.

The DataTurn Migration Process

Let’s go over the data conversion process from start to finish, and describe how it fits into the bigger picture of an industrialized migration process.

1. Step one is to export configuration files from source database and load those into DataTurn. DataTurn automatically creates target database structure based on what was in the legacy system.
2. Next, database mapping can be configured to normalize the data in the target, or to otherwise modify the target database to suit the customer’s needs. The I/O module is generated from this mapping.
3. DataTurn then generates the database schema and creates tables, indexes, and triggers in the target database; resulting in an empty target database.
4. Data from the source database is exported and then loaded into the target database.  Data is mapped and converted on the fly using data from the configuration process (step 2).

It’s important to understand that these steps all occur within the context of a larger industrialized migration process. TestMatch and DataMatch provide for highly automated testing processes, so that the product of code refactoring (from CodeTurn) and data conversion (from DataTurn) can be tested using an extraordinarily efficient, easily repeatable process.

If testing reveals any required changes, those changes can be implemented, followed by automated re-factoring by CodeTurn, automated data conversion by DataTurn, and more automated testing. In this way, it becomes possible to run test after test, based on actual user activity in the production system, as many times as you want, with very little additional time and effort. The processes of re-running data transfers, recording test scenarios, replaying those tests in the target system, and validating the results can be performed quickly and efficiently.  When the time comes to cut over to the target system, everything will have undergone thorough testing, and the team can move forward with complete confidence.

With the Migration Factory, we spend most of our effort working with clients to design and map the migration process. Whereas many traditional migration projects are executed in stages in order to mitigate risk, the Migration Factory enables a clean cutover of both code and data to a new target system, because the migrated system has been so thoroughly tested.

Astadia’s acquisition of Anubex combines our company’s industry leading expertise in mainframe modernization with a world-class toolset that industrializes the refactoring of legacy workloads, simplifies the migration of databases, and enables a holistic and automated approach to testing the end-to-end migration process.  For more information about the Astadia® Migration Factory™ , download the fact sheet, or contact us to schedule a meeting with one of Astadia’s Mainframe Migration experts.