Ahsay Upgrade Advisory (#34490) - Data corruption issue on any v9.1.2.17 to v9.1.4.21 backup types

Posted: 2022-06-02 Revised: 2022-06-27

We have recently identified and confirmed a critical bug found in AhsayCBS / OBM / ACB v9 (v9.1.2.17 to v9.1.4.21) that affects all types of Backup Sets, where source backup data is >32MB, resulting in unrestorable v9 data.

Not affected is restoring data from time period prior to v9.1.2.17 backup dates.

Not affected are Client Agents (OBM, ACB) running v8, or pre-v9.1.2.17.

• UPDATE: On 2022-June-27 AhsayCBS v9.3.2.0 is released; upgrade to this latest release to fix the issue.

• Affecting Ahsay v9.1.2.17 up to v9.1.4.21 , for ANY Backup Set types where source files have raw size greater-than 32MB, and Deduplication = OFF.
• Example, “tasks.zip” is 40MB, is affected.
• Example, “mylist.txt” is 2MB, is not affected.

• This happens in lookup (find duplicated data) logic in backup of all Backup Set types, when Deduplication is OFF.
• v9 will always find duplicated data from index even Deduplication is OFF. The lookup result will be wrong, hence, the file data is already corrupt.

• Unable to restore due to corrupt backup data

• Ahsay v9.1.2.17 up to v9.1.4.21 , where Deduplication = OFF at any point during backup runs within these versions
• Ahsay v9.1.2.17 up to v9.1.4.21 , where Deduplication = ON and Migrate Data = ENABLED (checkbox marked)
• Ahsay v9.1.2.17 up to v9.1.4.21 , where Deduplication = ON and Migrate Data = DISABLED (checkbox unmarked)
• [Updated: 2022-06-22] Additional analysis has shown that Backup Sets where Deduplication = ON (Enabled) , are not affected by this corruption.

• By default, Deduplication is ON (enabled), both module and Backup Set.
• Deduplication module is automatically assigned when account is created from CBS UI, and Deduplication toggle=ON when Backup Sets are created.
• Deduplication module is automatically assigned when CBS v7/v8 is upgraded to CBS v9, and Deduplication toggle=ON if In-File Delta was enabled in pre-v9 Backup Sets.
• It will be OFF (disabled) if either Deduplication toggle is OFF manually per Backup Set, by Policy, or Deduplication module is not enabled. If you use Ahsay API or third-party app to create or manage users, you should verify your behavior.
• Above are sample default behavior, as there can be other contributing factors that may affect settings, you should review your CBS and user settings to verify behavior in your environment.

• It is not affected. These source data isn't chunked for Deduplication.

• It depends on the conditions stated above.

• Yes.
• AhsayCBS V9.1.2.0 was the base release version

• Affected are those OBM you applied hotfix v9.1.2.17+ , but not other OBM/ACB running v9.1.2.0

• Pre-v9.1.2.17 data can be restored. Whether pre-v9.1.2.17 can be used depends on whether the backup data of pre-v9.1.2.17 is deleted by Space Free Up routine (Retention cleanup) or not. If pre-v9.1.2.17 data still exists, restoration can be done by revert index.

• No

• Yes, if you enabled AUA for users to upgrade to v9.1.4.0, and it meets the conditions stated above.
• Yes, if you have Agentless (Run on Server) jobs, and it meets the conditions stated above.
• No, if you kept AUA disabled, and users run pre-v9.1.2.17 client.

• Yes, if you enabled AUA for users to upgrade to v9.1.4.0, and it meets the conditions stated above.
• Yes, if you have Agentless (Run on Server) jobs, and it meets the conditions stated above.
• No, if you kept AUA disabled, and users run v8 client, they continue to follow In-File Delta settings.

• No workaround. Data must be backup again after fix is applied.

• Take immediate action to download and install latest release of AhsayCBS v9 and hotfix v9.1.4.22 (or higher) via Ahsay Partner Portal (https://www.ahsay.com/partners). The latest posted hotfix is bundled with Auto Update (AUA) binaries to push the client hotfix to OBM and ACB devices when CBS AUA feature is enabled per user.
  • UPDATE: On 2022-June-27 we released AhsayCBS v9.3.2.0; it is recommended to upgrade to this latest release as it includes prior hotfixes. Hotfixes for v9.1.4.x is no longer available.
• Once required minimum hotfix version is applied, as a one-time routine the next backup job will automatically first run a PDIC [corrected: 2022-06-21] will automatically remove Blocks (BAK) containing backup files that reference invalid checksum file chunks, then reupload source backup file that was corrupted and correctly update the migrate status table. A flag will be set in the Index to denote fix was applied ( likewise reverting the Index to an older copy will retrigger the fix ). [Shared Blocks (BAK), a block containing multiple small source files (valid and corrupt), won't be purged until all contents of shared block has cycled through changes and surpass Retention period.]
  • Showing when the fix was applied, you can review the next run's backup set log file, for entry “Run to remove backup files referencing dummy checksum file chunks for once…Completed”

• Stop! Do not upgrade until you contact a member our Sales team sales-kb@ahsay.com for assistance with your maintenance renewal.