Modernizing a Machine Scheduling System with Live Data Migration and CDC

Modernizing a Machine Scheduling System with Live Data Migration and CDC

Project Overview

Our team undertook the modernization of an existing machine-based time allocation and scheduling system using newer technologies and a more scalable architecture. The new web platform was developed using React for the frontend and ASP.NET Core for the backend.

Because the existing system was live, the modernization needed to preserve its core functionality while maintaining continuity with the existing system and providing a safe fallback strategy. This allowed the business to continue using the existing system if any issues were found in the new platform.

As part of the modernization, we redesigned the scheduling platform to provide a more intuitive web-based experience for managing machine jobs and workloads. Users can assign jobs to individual machines and manage those assignments through a visual calendar with monthly, weekly, and daily views. This provides a clear overview of job timelines, machine availability, and workloads.

We also designed and developed a new version of the existing mobile application specifically for operators. The mobile application provides operators with a more convenient way to access their assigned jobs and interact with the scheduling system while working on the machines. This extends the platform beyond the traditional web interface and provides operators with a solution better suited to their day-to-day workflow.

The platform supports multiple user roles, including operators, internal staff, and customers, allowing each user to access the functionality relevant to their responsibilities. The existing system also integrates with MYOB, and as part of the modernization, we enhanced the synchronization functionality to improve how operational and financial data is exchanged between the systems.

Another major part of the modernization was migrating the existing production data into a new database structure. Since the legacy system contained millions of records and was still actively used, the migration required careful planning to protect existing data, maintain backward compatibility, and ensure that changes made during the transition were not lost.

To support this process, we introduced a Change Data Capture (CDC) and SQL Server Integration Services (SSIS) based synchronization approach. CDC captured changes made to the live database, while SSIS was used to process, transform, and synchronize those changes between the legacy and new database structures during the migration.

Overall, the project involved more than simply rebuilding the existing system. It included modernizing the web platform, designing a new mobile experience for operators, introducing a new database structure, integrating modern technologies, and establishing a controlled migration and synchronization strategy for the existing live system.

The Challenge

The existing scheduling system was already being actively used in a live production environment and contained millions of records accumulated over time. Moving this data to a new database structure was therefore more complex than performing a standard database migration.

We needed to achieve several objectives simultaneously:

  • Preserve the existing production data through reliable backups.
  • Migrate millions of records into the new database structure.
  • Minimize disruption to the live system during the migration.
  • Ensure that new changes made to the existing system were not lost while the migration was taking place.
  • Maintain legacy system continuity, allowing clients to continue using the old system if a specific situation required it.
  • Keep the old and new database structures synchronized during the transition.
  • Provide a safe way to return to the existing system if critical issues were discovered in the new platform.

A traditional one-time data migration would not be sufficient. While the historical data could be copied to the new database, new records and updates could continue to occur in the live system during the migration. This could result in differences between the old and new databases.

We therefore needed a reliable mechanism to identify, process, and transfer changes occurring in the source database after the initial data migration.

The Solution

To address these requirements, we designed a migration and synchronization process using SQL Server Change Data Capture (CDC) and SQL Server Integration Services (SSIS).

The first step was to prepare a backup of the existing production data before making changes to the live environment. This provided a recovery point and helped protect the existing system during the modernization process.

We then migrated the existing data from the legacy database structure into the new database structure. Because the database contained millions of records, the migration had to be carefully planned and executed to avoid unnecessary impact on the live application.

The main challenge was handling changes that occurred while the migration was in progress.

This is where CDC and SSIS became important parts of the architecture.

Change Data Capture

SQL Server Change Data Capture (CDC) tracks changes made to the source database, including operations such as:

  • Insert
  • Update
  • Delete

Instead of repeatedly copying the entire database, CDC allowed us to identify the changes that occurred in the live system after the initial migration.

These captured changes could then be processed and transferred to the new database structure.

SSIS-Based Synchronization

We used SQL Server Integration Services (SSIS) to process and synchronize data between the legacy and new database structures.

The SSIS packages handled the data movement and transformation required between the different database structures. This was particularly important because the new database was not simply an identical copy of the old database. The data structures had been redesigned as part of the modernization.

The overall process allowed us to capture changes from the live system and apply the required transformations before synchronizing them with the new database.

This provided a controlled synchronization pipeline between the existing and modernized environments.

Two-Stage Migration Approach

This allowed us to separate the migration into two main stages:

  1. Initial Data Migration

Existing historical data was transferred from the old database structure into the new structure.

  1. Change Synchronization

CDC captured changes occurring in the live system, and SSIS processed and synchronized those changes with the new database.

This approach allowed the migration to continue while the existing system remained operational, significantly reducing the risk of data inconsistencies between the two environments.

Maintaining Legacy System Continuity

Another important requirement was that the old system could not simply be switched off immediately after the migration.

There may be situations where a client needs to access the legacy system for a specific operational requirement. More importantly, if a critical issue was discovered in the new platform, we needed the ability to move users back to the existing system while the issue was investigated and resolved.

For this reason, the migration strategy needed to keep the old and new environments synchronized during the transition.

CDC and SSIS provided an important part of this fallback strategy. Changes made in the live environment could continue to be captured and synchronized, allowing the legacy system to remain available without creating a separate, disconnected version of the data.

This meant that the modernization could be introduced gradually rather than requiring a single high-risk cutover.

If an issue occurred in the new system, the business could continue using the existing system while the development team worked on resolving the issue. Once the new platform was stable, the transition could continue.

This approach helped protect the live business process from unnecessary disruption during the modernization.

The New Scheduling Experience

Alongside the database modernization, we developed a new web-based scheduling experience using React and ASP.NET Core.

The React frontend provides an interactive interface for creating and managing jobs against individual machines through a visual calendar. Different calendar views provide visibility into workloads at different levels:

  • Monthly view – provides a high-level overview of machine workloads.
  • Weekly view – helps teams plan upcoming work.
  • Daily view – provides detailed visibility into individual machine assignments and timelines.

The ASP.NET Core backend provides the APIs and business logic required to manage scheduling data, users, machine assignments, and other system operations.

The system also provides role-based access for operators, internal staff, and customers, ensuring that users have access to the functionality relevant to their responsibilities.

Operator Mobile Application

As part of the modernization, we also designed and developed a new version of the existing mobile application for operators.

Instead of requiring operators to rely entirely on the traditional web application, the mobile application provides a more convenient way to access their assigned work and interact with the scheduling system while working on machines.

This extended the modernized platform beyond the desktop experience and provided an interface designed specifically around the operator workflow.

MYOB Integration

MYOB integration was already part of the existing system, so this was not a completely new integration introduced during the modernization.

As part of the new platform, we enhanced the existing MYOB integration by adding and improving synchronization functionality. This allowed the modernized system to continue working with the existing MYOB processes while providing more reliable data synchronization between the systems.

This approach allowed us to preserve the existing business processes while improving the way relevant operational and financial information was synchronized.

Technology Approach

The modernization combined several technologies to address different parts of the system:

  • React – modern web frontend and scheduling interface.
  • ASP.NET Core – backend APIs and business logic.
  • SQL Server – database platform for the modernized system.
  • CDC (Change Data Capture) – captures changes from the live legacy database.
  • SSIS (SQL Server Integration Services) – processes, transforms, and synchronizes data between the legacy and new database structures.
  • Mobile application – dedicated experience for operators.
  • MYOB synchronization – enhanced synchronization functionality between the scheduling platform and the existing MYOB integration.

Together, these technologies allowed us to modernize both the application layer and the underlying data architecture while maintaining a controlled transition from the existing live system.

The Results

The modernization has provided improvements across both the scheduling workflow and the underlying data architecture.

The new platform provides:

  • Centralized machine and job scheduling.
  • Improved visibility of machine availability and workloads.
  • Monthly, weekly, and daily scheduling views.
  • A dedicated mobile experience for operators.
  • Role-based access for different types of users.
  • Reduced scheduling conflicts through clearer job visibility.
  • Enhanced synchronization with MYOB while maintaining the existing integration.
  • A structured approach for migrating millions of existing records.
  • Backup and recovery considerations for the existing production data.
  • CDC-based change tracking from the legacy database.
  • SSIS-based data processing and synchronization.
  • Legacy system continuity during the transition period.
  • A fallback path to the existing system if issues are discovered in the new platform.

Most importantly, the system modernization could be performed without treating the existing live system as something that could simply be switched off.

By combining large-scale data migration, CDC-based change tracking, SSIS-based synchronization, and legacy system continuity, we were able to introduce the new database structure while continuing to support the existing production environment.

Looking Ahead

Modernizing a live system with millions of records is not simply a matter of creating a new database and moving the existing data into it. The migration must also account for the changes that continue to occur while the migration is running and the possibility that users may still need the legacy system.

For this project, CDC and SSIS were key components of the migration strategy. CDC allowed changes in the live database to be captured, while SSIS provided the data processing, transformation, and synchronization required to keep the legacy and new database structures aligned.

The fallback capability was equally important. Rather than making a single irreversible switch to the new platform, we could keep the existing system available while the new system was introduced and validated.

The result is a more modern and scalable scheduling platform while maintaining a controlled transition from the legacy system. This approach allowed us to modernize the underlying architecture without sacrificing data integrity, operational continuity, or the ability to continue using the existing system when required.

This website uses cookies.