User Tools

Site Tools


public:data_integrity_check

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
public:data_integrity_check [2020/04/27 18:29]
ronnie.chan
public:data_integrity_check [2020/10/15 09:59] (current)
yuk.cheng
Line 6: Line 6:
  
   * Bad program exits (i.e. AhsayCBS/​AhsayOBM/​AhsayACB application terminated unexpectedly when an active backup job is in progress)   * Bad program exits (i.e. AhsayCBS/​AhsayOBM/​AhsayACB application terminated unexpectedly when an active backup job is in progress)
 +
   * Technical problems on the AhsayOBM/​AhsayACB client machine (e.g. hardware failure, unexpected reboot, unexpected loss of power)   * Technical problems on the AhsayOBM/​AhsayACB client machine (e.g. hardware failure, unexpected reboot, unexpected loss of power)
 +
   * Technical problems on the AhsayCBS backup server (e.g. hardware failure, unexpected reboot, unexpected loss of power, storage issues, human error)   * Technical problems on the AhsayCBS backup server (e.g. hardware failure, unexpected reboot, unexpected loss of power, storage issues, human error)
 +
   * Technical problems on the cloud storage service   * Technical problems on the cloud storage service
  
Line 17: Line 20:
  
   * Identify and remove the files and/or folders in the backup destination(s) which do not appear in the index   * Identify and remove the files and/or folders in the backup destination(s) which do not appear in the index
 +
   * Identify and remove the files and/or folders which appear in the index but do not actually exist in the backup destinations (i.e. AhsayCBS, Cloud storage, or Local storage)   * Identify and remove the files and/or folders which appear in the index but do not actually exist in the backup destinations (i.e. AhsayCBS, Cloud storage, or Local storage)
 +
   * Identify and remove corrupted files from the backup destination(s) when the **Run Cyclic Redundancy Check (CRC) During Data Integrity Check** setting is enabled   * Identify and remove corrupted files from the backup destination(s) when the **Run Cyclic Redundancy Check (CRC) During Data Integrity Check** setting is enabled
 +
   * Identify and remove partially uploaded (orphan) files from the backup destination(s) to free up storage space   * Identify and remove partially uploaded (orphan) files from the backup destination(s) to free up storage space
 +
 +  * identify and remove any index files which are more than 90 days old from the backup destination(s)
 +
   * **(TEST MODE) confirmation screen** (applicable on AhsayOBM/​AhsayACB client)   * **(TEST MODE) confirmation screen** (applicable on AhsayOBM/​AhsayACB client)
 +
   * Update storage statistics   * Update storage statistics
  
Line 32: Line 42:
  
   * AhsayOBM/​AhsayACB client GUI   * AhsayOBM/​AhsayACB client GUI
 +
   * AhsayCBS Web Console for Run on Server (Office 365 and Cloud File) Backup   * AhsayCBS Web Console for Run on Server (Office 365 and Cloud File) Backup
 +
   * //​RunDataIntegrityCheck.bat//​ batch file (applicable for Windows operating system only)   * //​RunDataIntegrityCheck.bat//​ batch file (applicable for Windows operating system only)
 +
   * //​RunDataIntegrityCheck.sh//​ script file (applicable for FreeBSD/​Linux (CLI) operating systems only)   * //​RunDataIntegrityCheck.sh//​ script file (applicable for FreeBSD/​Linux (CLI) operating systems only)
  
Line 39: Line 52:
  
 There are two (2) data integrity check modes: There are two (2) data integrity check modes:
 +
   * With Run Cyclic Redundancy Check (CRC) disabled (Default mode)   * With Run Cyclic Redundancy Check (CRC) disabled (Default mode)
 +
   * With Run Cyclic Redundancy Check (CRC) enabled   * With Run Cyclic Redundancy Check (CRC) enabled
 +
 +<WRAP indent>
  
 ==== With Run Cyclic Redundancy Check (CRC) disabled (Default mode) ==== ==== With Run Cyclic Redundancy Check (CRC) disabled (Default mode) ====
 +
 This is the default setting of the data integrity check. Running a data integrity check on this mode allows the AhsayOBM/​AhsayACB client or AhsayCBS backup server to perform a comparison between the files and/or folders on the backup destination(s) and the list of the files and/or folders recorded in the current index file. This is the default setting of the data integrity check. Running a data integrity check on this mode allows the AhsayOBM/​AhsayACB client or AhsayCBS backup server to perform a comparison between the files and/or folders on the backup destination(s) and the list of the files and/or folders recorded in the current index file.
  
-{{public:​dic-v8-01.png}}+{{public:​dic-v8-01a.png}}
  
 <WRAP info> <WRAP info>
Line 76: Line 94:
  
   * When a backup destination has encountered a hardware failure (e.g. a disk failure on an AhsayCBS user home drive or AhsayOBM/​AhsayACB Local destination drive).   * When a backup destination has encountered a hardware failure (e.g. a disk failure on an AhsayCBS user home drive or AhsayOBM/​AhsayACB Local destination drive).
 +
 +</​WRAP>​
  
 </​WRAP>​ </​WRAP>​
Line 83: Line 103:
  
 </​WRAP>​ </​WRAP>​
 +
 +===== Limitations =====
 +
 +  * Data Integrity Check has to be started manually from the AhsayOBM/​AhsayACB client UI. It cannot be remotely started from the AhsayCBS web console or scheduled backup to run automatically. The only exception is for a Run on Server (Office 365 or Cloud File) backup sets were a data integrity check can be started from the AhsayCBS web console
 +
 +  * When a Data Integrity Check has identified issues on the backup set, it may require the end user to confirm the changes before it takes the corrective actions
 +
 +  * When a data integrity check is running, a backup and restore job cannot be run and vice versa: When an active backup or restore job(s) is running, a data integrity check cannot be run
 +
 +===== How It Works =====
 +
 +The following diagrams show the detailed flow for each data integrity check mode.
 +
 +==== With Run Cyclic Redundancy Check (CRC) disabled (Default mode) ====
 +
 +{{public:​dic-v8-03.png}}
 +
 +==== With Run Cyclic Redundancy Check (CRC) enabled ====
 +
 +{{public:​dic-v8-05.png}}
 +
 +===== Test Mode Confirmation Screen =====
 +
 +Normally aspart of the data integrityjob,​ **(TEST MODE)** confirmation screen is usually displayed once a data integrity check is completed, which gives a summary report of the corrupted files, invalid indexes,or storage statistics issue for each backup destination. The (TEST MODE) confirmation screen allows the enduser to review the results of the data integrity check, and to decide whether they would like to proceed with the corrective actions.To further streamline the data integrity check process and improve user experience,​the (TEST MODE) confirmation screen will **ONLY** prompt if either of the <color #​ed1c24>​criteria’s</​color>​ below matches the backup data during the data integrity check operation:
 +
 +  * deletednumber of backupfiles isover 1,000
 +
 +  * deletednumber of backup file size isover 512MB (in total)
 +
 +  * deletednumber of backup files isover 10% ofthetotal backup files
 +
 +Otherwise, the data integrity check job will **automatically** take corrective actions.
 +
 +The (TEST MODE) screen includes five (5) summary report for the following items found per backup destination:​
 +
 +^ Items ^ Description ^
 +^ Destination | this indicates the destination of the backup data where the data integrity check will be run |
 +^ Items found in index | the number of files and its total size (MB) that appear in the index |
 +^ Data corrupted items | the number of files and its total size (MB) found to be corrupted |
 +^ Index broken data blocks | index with its associated data blocks which found to be corrupted |
 +^ Statistics | storage statistics status of the data area and retention area of the backup destination (i.e. correct or incorrect status) |
 +
 +<WRAP info>
 +Although you select ALL backup sets before starting the data integrity check, the (TEST MODE) confirmation screen will prompt one at a time with the corresponding backup set(s). \\ \\ For example, the data integrity check has run with three (3) backup sets and all these backup sets match the criteria’s of the (TEST MODE) confirmation screen, the (TEST MODE) confirmation screen will prompt three times to confirm if the end user will take corrective actions for the three backup sets.
 +</​WRAP>​
 +
 +Below is an example of a (TEST MODE) confirmation screen with the following scenario:
 +
 +  * Multiple backup destinations,​ corrupted items and index-related issues found with correct and incorrect storage statistics. \\ {{public:​dic-v8-04.png}}
 +
 +===== How does Data Integrity Check (DIC) compare with Periodic Data Integrity Check (PDIC) =====
 +
 +Periodic data integrity check is performed at the beginning of a backup job, which provides an additional regular data integrity check of the backup data and updates the storage statistics for each backup set. The PDIC feature is enabled from v8.3.2.11 or above and cannot be turned off. This is to ensure a maximum protection of the backup data. 
 +
 +Unlike with the Data Integrity Check (DIC), the PDIC starts automatically and performs a quick check of all the backup destination(s) without the end user intervention. ​
 +
 +The PDIC will be initiated automatically once **EITHER** of the following conditions is met:
 +
 +  * Will be triggered on a weekly basis, usually on the first run of backup job that falls on any of these days: Friday, Saturday, or Sunday
 +    ​
 +  * If there is no active backup job(s) running on Friday, Saturday, or Sunday, then the PDIC will be triggered on the next available backup job
 +
 +E.g. If the last PDIC job was run more than seven (7) days ago, then the subsequent PDIC job(s) will run seven days from that day onwards.
 +
 +==== Comparison ====
 +
 +^ Features ^ Data Integrity Check (DIC) ^ Periodic Data Integrity Check (PDIC) ^
 +^ Runs automatically | <color #​ed1c24>​X</​color>​ | ✓ |
 +^ Allows selection of backup destination(s) | ✓ | <color #​ed1c24>​X</​color>​ |
 +^ Run Cyclic Redundancy Check (CRC) feature | ✓ | <color #​ed1c24>​X</​color>​ |
 +^ Identify and remove the files and/or folders in the backup destination(s) which do not appear in the index | ✓ | ✓ |
 +^ Identify and remove the files and/or folders which appear in the index but do not exist in the backup destination(s) | ✓ | <color #​ed1c24>​X</​color>​ |
 +^ Identify and remove partially uploaded (orphan) files from the backup destination(s) | ✓ | <color #​ed1c24>​X</​color>​ |
 +^ (TEST MODE) confirmation screen feature | ✓ | <color #​ed1c24>​X</​color>​ |
 +^ Update Storage Statistics | ✓ | ✓ |
  
 ===== FAQs ===== ===== FAQs =====
  
-  * [[public:​5004_how_to_run_a_data_integrity_check|How to run a Data Integrity Check for backup data stored in backup destination]] 
   * [[public:​8008_faq:​how_to_run_a_data_integrity_check_on_linux_cli|How to run a Data Integrity Check on Linux (CLI) machine]]   * [[public:​8008_faq:​how_to_run_a_data_integrity_check_on_linux_cli|How to run a Data Integrity Check on Linux (CLI) machine]]
  
public/data_integrity_check.1587983362.txt.gz · Last modified: 2020/04/27 18:29 by ronnie.chan