====== Ahsay Upgrade Advisory (#34490) - Data corruption issue on any v9.1.2.17 to v9.1.4.21 backup types ====== **Posted:** 2022-06-02 **Revised:** 2022-06-27 We have recently identified and confirmed a critical bug found in AhsayCBS / OBM / ACB v9 (v9.1.2.17 to v9.1.4.21) that affects all types of Backup Sets, where source backup data is >32MB, resulting in unrestorable v9 data. Not affected is restoring data from time period prior to v9.1.2.17 backup dates. Not affected are Client Agents (OBM, ACB) running v8, or pre-v9.1.2.17. * **UPDATE: On 2022-June-27 AhsayCBS v9.3.2.0 is released; upgrade to this latest release to fix the issue.** ===== What does this mean? ===== * Affecting Ahsay v9.1.2.17 up to v9.1.4.21 , for **ANY Backup Set types where source files have raw size greater-than 32MB**, and Deduplication = OFF. * Example, "tasks.zip" is 40MB, is affected. * Example, "mylist.txt" is 2MB, is not affected. ===== What happened? ===== * This happens in lookup (find duplicated data) logic in backup of all Backup Set types, when Deduplication is OFF. * v9 will always find duplicated data from index even Deduplication is OFF. The lookup result will be wrong, hence, the file data is already corrupt. ===== What can the user expect? ===== * Unable to restore due to corrupt backup data ===== What are the affected Ahsay versions? ===== * Ahsay v9.1.2.17 up to v9.1.4.21 , where **Deduplication = OFF** at any point during backup runs within these versions * ~~Ahsay v9.1.2.17 up to v9.1.4.21 , where **Deduplication = ON and Migrate Data = ENABLED** (checkbox marked)~~ * ~~Ahsay v9.1.2.17 up to v9.1.4.21 , where **Deduplication = ON and Migrate Data = DISABLED** (checkbox unmarked)~~ * [Updated: 2022-06-22] Additional analysis has shown that Backup Sets where **Deduplication = ON (Enabled)** , are __not affected__ by this corruption. ===== What is the default Deduplication settings? ===== * By default, Deduplication is ON (enabled), both module and Backup Set. * Deduplication module is automatically assigned when account is created from CBS UI, and Deduplication toggle=ON when Backup Sets are created. * Deduplication module is automatically assigned when CBS v7/v8 is upgraded to CBS v9, and Deduplication toggle=ON if In-File Delta was enabled in pre-v9 Backup Sets. * It will be OFF (disabled) if either Deduplication toggle is OFF manually per Backup Set, by Policy, or Deduplication module is not enabled. If you use Ahsay API or third-party app to create or manage users, you should verify your behavior. * Above are sample default behavior, as there can be other contributing factors that may affect settings, you should review your CBS and user settings to verify behavior in your environment. ===== What if the user's Source data is less-than 32MB? ===== * It is not affected. These source data isn't chunked for Deduplication. ===== What if the user's Source data to backup is a mix of greater-than and less-than 32MB? ===== * It depends on the conditions stated above. ===== v9.1.2.17+ was a hotfix, correct? ===== * Yes. * AhsayCBS V9.1.2.0 was the base release version ===== My AhsayCBS is v9.1.2.0, but I applied hotfix to a few AhsayOBM, who is affected? ===== * Affected are those OBM you applied hotfix v9.1.2.17+ , but not other OBM/ACB running v9.1.2.0 ===== If backup data is corrupt, how about data pre-v9.1.2.17? Can that be restored? ===== * Pre-v9.1.2.17 data can be restored. Whether pre-v9.1.2.17 can be used depends on whether the backup data of pre-v9.1.2.17 is deleted by Space Free Up routine (Retention cleanup) or not. If pre-v9.1.2.17 data still exists, restoration can be done by [[https://wiki.ahsay.com/doku.php?id=public:version_9:general:9205_how_to_revert_index_snapshot|revert index]]. ===== Can I undo the v9.1.2.17+ hotfix and return to v9.1.2.0? ===== * No ===== I just upgraded from v9.1.2.0, to AhsayCBS v9.1.4.0, am I affected? ===== * Yes, if you enabled AUA for users to upgrade to v9.1.4.0, and it meets the conditions stated above. * Yes, if you have Agentless (Run on Server) jobs, and it meets the conditions stated above. * No, if you kept AUA disabled, and users run pre-v9.1.2.17 client. ===== I just upgraded from v8, to AhsayCBS v9.1.4.0, am I affected? ===== * Yes, if you enabled AUA for users to upgrade to v9.1.4.0, and it meets the conditions stated above. * Yes, if you have Agentless (Run on Server) jobs, and it meets the conditions stated above. * No, if you kept AUA disabled, and users run v8 client, they continue to follow In-File Delta settings. ===== Is there workaround? ===== * No workaround. Data must be backup again after fix is applied. ===== What action do I need to take to fix this problem? ===== * **Take immediate action** to download and install latest release of **AhsayCBS v9** ~~and **hotfix v9.1.4.22 (or higher)** via Ahsay Partner Portal (https://www.ahsay.com/partners)~~. ~~The latest posted hotfix is bundled with Auto Update (AUA) binaries to push the client hotfix to OBM and ACB devices when CBS AUA feature is enabled per user.~~ * **UPDATE: On 2022-June-27 we released AhsayCBS v9.3.2.0; it is recommended to upgrade to this latest release as it includes prior hotfixes. Hotfixes for v9.1.4.x is no longer available.** * Once required minimum hotfix version is applied, as a one-time routine the next backup job ~~will automatically first run a PDIC~~ [//corrected: 2022-06-21//] will automatically remove Blocks (BAK) containing backup files that reference invalid checksum file chunks, then reupload source backup file that was corrupted and correctly update the migrate status table. A flag will be set in the Index to denote fix was applied ( likewise reverting the Index to an older copy will retrigger the fix ). [Shared Blocks (BAK), a block containing multiple small source files (valid and corrupt), won't be purged until all contents of shared block has cycled through changes and surpass Retention period.] * Showing when the fix was applied, you can review the next run's backup set log file, for entry "//Run to remove backup files referencing dummy checksum file chunks for once...Completed//" ===== What if my maintenance has already expired? How do I upgrade? ===== * **Stop! Do not upgrade** until you contact a member our Sales team [[mailto:sales-kb@ahsay.com|sales-kb@ahsay.com]] for assistance with your maintenance renewal.