To reduce the amount of cost-expensive operations incurred by your cloud storage provider and to decrease the amount of traffic being sent over the network when moving or copying data to object storage, Veeam Backup & Replication uses indexes.
Indexes behavior is as follows:
- Indexes are created (or updated) during each offload or copy session and consist of hash values of blocks that are being transferred to object storage. These hashes are retrieved from meta information of your backup files (.vbk, .vib, or .vrb).
- Indexes are stored in the ArchiveIndex directory that is located on the source extent from which the backup data was offloaded or copied.
On each subsequent offload/copy session, Veeam Backup & Replication reuses these indexes to verify whether new blocks that are about to be transferred to object storage have not been offloaded earlier. Verification is done by comparing existing indexes hashes with that of a block being transferred.
- Indexes are built per backup chain and cannot have any cross references to any other backup chains.
- Indexes are updated every time a backup chain is modified.
For example, some data might have been removed due to the retention policy threshold, or you may have removed it manually. Both scenarios will modify your indexes upon the next successful offload or copy session to maintain consistency.
- Corrupted indexes can be rebuilt anew by using the Rescan feature, as described in Rescanning Scale-Out Repositories.
Once rebuilt, Veeam Backup & Replication will have to wait for 24 hours before it can offload any data again. This is done to comply with the Eventual Consistency model of Amazon S3.
ArchiveIndex Directory Structure
When Veeam Backup & Replication creates indexes, it also creates and maintains the following directory structure on each extent.
The root directory for keeping indexes.
Contains objects in a backup file.
An identifier of an object in a backup file.
Contains indexes of offloaded backup files (.vbk, .vib, or .vrb).
Contains meta information on hash values stored in index files.