User Tools

Site Tools


public:new-index-v8:new-processes

New Index and Backup Processes Introduced in Ahsay v8.3

Four new processes were introduced in the Ahsay Backup Software v8.3.0.0 (or above) to provide better backup performance, robustness and data integrity:

1. New Index Implementation

Ahsay Backup Software v8.3 has a new index design to provide better backup performance and robustness. However, for customers who are going to upgrade from older versions (v6, v7, and pre-v8.3), they still need to undergo a mandatory index conversion process before the new index can be used and normal backup jobs can continue.

After the v6, v7 and pre-v8.3 AhsayOBM and AhsayACB clients are upgraded to v8.3, the backup set index conversion will occur immediately on the first run of backup job.

1.1 Index Conversion

For AhsayOBM/AhsayACB client

v6 backup sets:

Index.bdb and r-index.bdb will be converted to the new index file structure: index.db* and backupInfo.db*

1. Old index file (index.bdb) will be converted to the new index file structure: (index.db)

2. Data Integrity Check will be performed once the index conversion is finished

v7 and pre-v8.3 backup sets:

index.b2b*, index.xml* and index-s0 will be converted to the new index file structure: index.db* and backupInfo.db*

1. Old index file (index.b2b) will be converted to the new index file structure: (index.db)

2. Data Integrity Check will be performed once the index conversion is finished

For AhsayCBS Run on Server (Office 365 and Cloud File) backup

v7 and pre-v8.3 backup sets:

index.b2b*, index.xml* and index-s0 will be converted to the new index file structure: index.db* and backupInfo.db*

1. Old index file (index.b2b) will be converted to the new index file structure: (index.db)

2. Data Integrity Check will be performed once the index conversion is finished

1.2 Index Conversion requirements

For AhsayOBM/AhsayACB client

Client Version Index Conversion Needed Data Migration Needed
v6
v7 X
Earlier than v8.3.0.0 X
v8.3.0.0 or above X X

Please note:

  • Index conversion cannot be disabled.
  • For run on client backup sets, index conversion process will automatically start during the first backup job after upgrading AhsayOBM/AhsayACB to v8.3.0.0 or above.
  • After index conversion, the size of the index will not be the same. For large data index, the new index will become smaller since duplicate information will be grouped. While for small data index, the new index may be bigger since additional information may be

For AhsayCBS Run on Server (Office 365 and Cloud File) backup

Client Version Index Conversion Needed Data Migration Needed
v7 X
Earlier than v8.3.0.0 X
v8.3.0.0 or above X X

Please note:

  • For run on server backup sets (Office 365 and Cloud File), index conversion process will automatically start during the first backup job after upgrading AhsayCBS to v8.3.0.0 or above.

For backup sets which contain large number of files and/or folders, the v8.3 index conversion process could take several hours to complete. For example: File, Cloud File, MS Exchange mail level, and Office 365 backup sets. In some cases, backup sets containing several million files and/or folders could take days to complete the v8.3 index conversion process. Kindly take this into consideration when planning your AhsayOBM/AhsayACB client upgrade to v8.3.0.0 or above.

2. Backup Set Index Handling Process

The main purpose of the new backup set index handling process is to provide an improved backup user experience by automating the handling of corrupted index related issues. Therefore, it can be guaranteed that the backup job process will NOT be interrupted or prevented from running when index related issues are encountered.

This whole process is done without the need for intervention by the end customer or backup service provider, which reduces the overall cost of support.

The new backup set index handling process will perform the following:

1. All available index files (index.db) from the current directory (e.g. cloud destination, AhsayCBS, local drive, and FTP or SFTP) will be sorted according to the modified date and time.

2. If the local temporary directory and backup job folder also contain index files, then these index files will also be added to the list.

3. Latest index file in the list will be downloaded according to the most current modified date and time.

4. The latest modified index file will be verified if it is valid and can be opened.

5. If it CAN be opened, then it will be used to compile file list for backup.

6. If NOT, then the index handling process will proceed to the next index file from the list (sorted by latest modified time to oldest) until a valid index is found that can be compiled for backup.

Corrupted index files will no longer be recoverable. Corrupted index files will remain in the backup destination(s) and can only be removed by manually running the Data Integrity Check on the AhsayOBM/AhsayACB client or AhsayCBS Web Console for Run on Server (Office 365 and Cloud File) backups. It is advisable to perform a Data Integrity Check regularly to ensure that NO corrupted data will remain in the backup destination(s).

3. Post-Backup Data Validation Check

Although as part of the current backup process, a checksum validation is already performed on the backup files to ensure data integrity and recoverability, in addition, to provide peace of mind, an additional post-backup data validation check is now performed in v8.3.

The post-backup data validation check will perform the following:

1. The number of 16 or 32 MB data blocks in the backup destination(s) is identical to the number of blocks transferred

2. The individual sizes of each data block in the backup destination(s) is identical to the sizes of each block transferred

After performing a backup job, click the button to display the detailed backup log. This will show if the post-backup data validation check has completed successfully.

The screenshot below shows an example of the post-backup data validation check process in a backup job.

4. Periodic Data Integrity Check (PDIC)

Data integrity is a core focus of the Ahsay backup software. It is important to maintain the overall accuracy, completeness, and consistency of the backup data throughout its lifecycle. To ensure that these objectives are met, Periodic Data Integrity Check (PDIC) is implemented which performs automated error-checking and validation procedures throughout a backup job process.

Also, as part of the PDIC, the storage statistics for the backup set are recalculated to maintain up-to-date and accurate data usage which guarantees consistent customer billing experience.

During a backup job, the PDIC is performed to ensure the validity and recoverability of the backup data.

This is how the periodic data integrity check (PDIC) will run:

1. At the start of the backup job, the PDIC will check for the physical data blocks (.bak files) in the backup destination(s)

2. If there are physical data blocks (.bak files) found which do not exist in the index (i.e. data blocks with NO related index), then these physical data blocks will be automatically removed from the backup destination(s)

3. The statistics on the Data area and Retention area are recalculated

After the backup job, click the button to display the detailed backup log. This will show if the PDIC had run and completed successfully. There are two possible outcomes in performing the PDIC:

a. The PDIC job is completed without data blocks related issues encountered

b. The PDIC job has detected data blocks with NO related index

Result 1

The screenshot below shows an example of a backup job where the PDIC has run and has NOT detected any issues on the physical data blocks.

Result 2

The screenshot below shows an example of a backup job where the PDIC has run and has identified data blocks (.bak files) which do not exist in the index file. These physical data blocks are automatically deleted from the backup destination(s).

The PDIC only checks for the physical data blocks (.bak files) directly, instead of checking each file in the index, so if there are index files found to be corrupted, then these index files and its associated data blocks can only be removed from the backup destination(s) by manually running the Data Integrity Check on the AhsayOBM/AhsayACB client or AhsayCBS Web Console for Run on Server (Office 365 and Cloud File) backups. It is advisable to perform a Data Integrity Check regularly to ensure that NO corrupted data will remain in the backup destination(s).

What conditions will trigger a Periodic Data Integrity Check

During a backup job, the Periodic Data Integrity Check (PDIC) operation will be triggered under EITHER of the following conditions:

  • Will be triggered on a weekly basis, usually on the first run of backup job that falls on any one of these days: Friday, Saturday, or Sunday
  • If there is no active backup job(s) running on Friday, Saturday, or Sunday, then the PDIC will be triggered on the next available backup job

    e.g. If the last PDIC job was run more than 7 days ago, then the subsequent PDIC job(s) will run 7 days from that day onwards.

Which types of backup jobs will initiate a Periodic Data Integrity Check

Periodic Data Integrity Check (PDIC) can be initiated by the following types of backup job:

  • Manual backup job on the AhsayOBM/AhsayACB client
  • Manual backup job on the AhsayCBS Web Console for Run on Server (Office 365 and Cloud File)
  • Scheduled backup job on the AhsayOBM/AhsayACB client
  • Scheduled backup job on the AhsayCBS Web Console for Run on Server (Office 365 and Cloud File)
  • Continuous backup job on the AhsayOBM/AhsayACB client
  • Continuous backup job on the AhsayCBS Web Console for Run on Server (Office 365 and Cloud File)
  • Windows System Tray Icon backup job
  • Windows Logout Reminder-initiated backup job
  • RunBackupSet.bat script (only applicable for Windows platform)
  • RunBackupSet.sh script (only applicable for Linux/FreeBSD platform)
  • AhsayCBS Web Console server-initiated backup job
public/new-index-v8/new-processes.txt · Last modified: 2020/04/28 15:31 by ronnie.chan

Page Tools