Avamar is an end-to-end backup solution consisting of hardware and software. It provides source-based deduplication which is global in that data found on several clients gets sent over the wire only once. Avamar uses Variable Block Granularity deduplication, which is an intelligent scanninga nd segmentation of data along logical boundries. It uses RAID, RAIN, replication and checkpoints for systematic fault tolerance, and uses standard IP connectivity in a lightweight fashion so no dedicated backup network is required. It is scalable by adding shelves of storage and supports a large number of client OS and applicatins
Terminology:
An object is a variable-sized unit of deduplicated data, also called a "chunk."
A stripe is a unit of disk space, or rather a "file" that stores objects.
A node is an Avamar server component running SuSE linux. There are utility nodes that manage connections and system operations and storage nodes that simply put data to disk and handle the deduplication.
A server is all Avamar nodes and the network, also called a "grid."
A system is the Avamar server and all clients.
Backup and restore are standard terminology in that they are a point-in-time copy of data and the returning of that data, respectively.
Initialization is the first backup. Typically takes longer due to seeding of data.
Transmission encryption on the Avamar is self-explanatory and can be configured for none, medium or high. See the Avamar documentation for current encryption levels and definition. Avamar will also encrypt data at rest using the same encryption protocols.
Retention and Replication are also standard terms in backup common to Avamar.
Avamar Administrator is the main configuration tool for Avamar systems. The Avamar Enterprise Administraotr is not the same product as the Data Domain Enterprise Administrator, and gives a less feature-rich subset of configuration but gives teh ability to review multiple Avamar systems in one spot.
Avamar's PostgreSQL database stores data relating to the system, is read-only but queriable for reporting. Avamar also provides a command-line-interface along with custom commands and scripts for managing and configuring the Avamar system.
Saturday, November 2, 2013
Friday, November 1, 2013
Data Domain License Options
Licensed options for Data Domain include:
- DD Boost - software client to distribute deduplication
- DSP - distributed segment processing takes the initial creation of fingerprint to the client, which then requests the vector summary from the DD appliance and sends the changed data. It's the same process with different location for the initial creation and verification
- Managed File replication - allows the application to determine that there is a replicated, deduplicated file elsewhere
- Creates its own storage unit in /col1/
- Advanced Load Balancing and link failure transparently fails backup off failed NIC to the remaining NICs in the system
- VTL - Virtual Tape Library
- DD OS 5.0 or newer only
- NAS backup through NDMP tape servers, which is NDMP over Ethernet
- IBMi - addon for VTL to accommodate IBM iSeries systems version 5.4, 6.1 and 6.1.1
- DD Replicator - asynchronous IP replication
- Encrypted replication uses SSL, AES 256 for software encryption of data in transit
- Encryption of data at rest is also software based and done after data is written, so a slight performance hit is taken
- Encrypts data in-line, so there is never unencrypted data on the system
- Encryption is all contained within the DD system, so it is transparent to systems connecting to the DD
- RSA DPM is also supported for key rotation as of DD OS 5.2
- DD Retention Lock
- Files can not be deleted until the retention period expires
- Includes file shredding
- Applied by using a policy with a minimum age of 12 hours and a mazimum age of 5 years
- Two options:
- Governance - retention lock, shredding
- Compliance - higher standard and more configurable, available on all DD except the GDA
Data Domain System Management
Data Domain uses an application called "Enterprise Manager" to administer, monitor and configure the DD appliance. Several appliances can be added to a single Enterprise Manager and administered as one system. The Data Domain Management Framework contains the CLI that can be accessed via SSH or at the console.
DD Management Framework contains the CLI, alerts and monitoring either through SNMP or email, RBAC and autosupport, which is EMC-provided 24/7. The DD also uses a Lights Out management, which gives the user the ability to power cycle and connect to the DD appliance whether or not the unit is running. DDOS 5.0 or later has this feature, which includes power management as well as serial over Ethernet (or serial over LAN).
DD Management Framework contains the CLI, alerts and monitoring either through SNMP or email, RBAC and autosupport, which is EMC-provided 24/7. The DD also uses a Lights Out management, which gives the user the ability to power cycle and connect to the DD appliance whether or not the unit is running. DDOS 5.0 or later has this feature, which includes power management as well as serial over Ethernet (or serial over LAN).
Data Domain Extended Retention
Data Domain Extended Retention is the new name for Data Domain Archiver. It is a licensed software feature which supports VTL and MTree, as well as CIFS and NFS data. It also utilizes "Fast Seeding" which improves data migration from the active to retention tiers of storage, and "packing" which reclaims duplicate segments in the retention tier. Deduplication takes place only within a specific tier, not across active and retention tiers.
Extended Retention is supported only on the DD860 and DD990 and with the ES20 or ES30 expansion cabinets.
The benefits of ER are that data can be written to the active tier very quickly and then moved off to slower retention tiers as time permits. The active tier will typically hold data for 90 days or less, with the retention tiers holding data as long as policy requires. There is also a high data ingress rate (up to 31TB/hour), and the cost per GB goes down drastically as the system scales with more expansion shelves.
The retention tier is made up of several retention units. Each unit can span multiple shelves if configured to do so, or can be portions of shelves. Data moves out of the active tier into the first retention unit until that unit is completely full. Once full, the unit is "sealed" and the second unit is started until it is full, etc. This way, oldest data is in one retention unit, followed by the next and the next. Deduplication takes place only in each unit to reduce the likelihood of losing multiple units should there be a failure. This is the Fault Isolation of Data Domain, as well.
All data on a DD Extended Retention system appears active. Tiering is transparent. To accommodate scalability, not all retention units are readily available in memory. Oldest data will be de-prioritized and put on standby but is still more accessible than tape. Data can also be replicated from one DD ER unit to another using collection replication, which replicates the entire box with each tier in place.
Extended Retention is supported only on the DD860 and DD990 and with the ES20 or ES30 expansion cabinets.
The benefits of ER are that data can be written to the active tier very quickly and then moved off to slower retention tiers as time permits. The active tier will typically hold data for 90 days or less, with the retention tiers holding data as long as policy requires. There is also a high data ingress rate (up to 31TB/hour), and the cost per GB goes down drastically as the system scales with more expansion shelves.
The retention tier is made up of several retention units. Each unit can span multiple shelves if configured to do so, or can be portions of shelves. Data moves out of the active tier into the first retention unit until that unit is completely full. Once full, the unit is "sealed" and the second unit is started until it is full, etc. This way, oldest data is in one retention unit, followed by the next and the next. Deduplication takes place only in each unit to reduce the likelihood of losing multiple units should there be a failure. This is the Fault Isolation of Data Domain, as well.
All data on a DD Extended Retention system appears active. Tiering is transparent. To accommodate scalability, not all retention units are readily available in memory. Oldest data will be de-prioritized and put on standby but is still more accessible than tape. Data can also be replicated from one DD ER unit to another using collection replication, which replicates the entire box with each tier in place.
Data Domain Systems Overview
As model of DD increases, so do ingest rate and capacity.
Logical capacity is the amount of data, based on standard set of calculations that can be backed up on a DD appliance, while usable capacity is the actual disk space available.
DD hardware follows N+1 philosophy, where hot-swappable components such as fans, power supplies, drives have an available spare in the event of a failure. Currently drive sizes are 1 or 2 TB, 7200 RPM SATA II.
The ES20 expansion shelf has 4 SAS connectors, or 2 for each module. 1 connector goes to the host and the other to another expansion shelf. The ES30 is the same as the ES20, but consumes 45% less power. For either expansion shelf, a "shelf capacity" license and DD OS 5.1 or newer is required.
The DD16 and DD640 do not support expansion shelves, and are internal storage capacity only. The 640 and 670 have internal storage while supporting optional external storage, and the 860, 890 and 99 are all external-only storage.
Logical capacity is the amount of data, based on standard set of calculations that can be backed up on a DD appliance, while usable capacity is the actual disk space available.
DD hardware follows N+1 philosophy, where hot-swappable components such as fans, power supplies, drives have an available spare in the event of a failure. Currently drive sizes are 1 or 2 TB, 7200 RPM SATA II.
The ES20 expansion shelf has 4 SAS connectors, or 2 for each module. 1 connector goes to the host and the other to another expansion shelf. The ES30 is the same as the ES20, but consumes 45% less power. For either expansion shelf, a "shelf capacity" license and DD OS 5.1 or newer is required.
The DD16 and DD640 do not support expansion shelves, and are internal storage capacity only. The 640 and 670 have internal storage while supporting optional external storage, and the 860, 890 and 99 are all external-only storage.
Data Domain Replication
Data Domain replication involves copying data from one DD to another. A replication pair is called a "context" in the DD GUI. The primary purpose of DD replication is disaster recovery, although archiving data for longer periods than is desired to have onsite is also a function.
There are 4 types of replication in Data Domain:
Collection replication is a full-system mirror of the col1 directory. The destination is read-only, and can back up one source only. Collection replication backups up snapshots as well as users and permissions, and is the fastest and lightest replication method of replicating one DD to another.
Directory replication is more flexible in its ability to configure replicated data from one DD to point to multiple destinations and selecting just the data to be replicated. CIFS and NFS shares can be sent to the same DD destination, but not within the same directory. When directory replication is configured, the destination folder will be automatically created if it does not exist, and after created ownership and permissions on files and folders is maintained. The directory replication process is triggered when a file closes, but to conserve high-demand bandwidth can be forced to a specific time. Snapshots are not replicated automatically and must be configured separately.
MTree replication is a new feature with the introduction of MTrees. In this method, the destination MTree is set read only, and the main difference is in how data is determined for transfer. MTree replication still sends changed data only, but instead of sending vector summaries back and forth, MTree replication creates a snapshot and compares one to another. MTree replication can use encryption and Retention Lock as well.
Pool replication is exactly like directory replication, but the source is VTL data. A separate VTL license is not needed onthe destination DD if VTL restores will not be done directly from it.
DD systems can replicate in the following topologies:
There are 4 types of replication in Data Domain:
Collection replication is a full-system mirror of the col1 directory. The destination is read-only, and can back up one source only. Collection replication backups up snapshots as well as users and permissions, and is the fastest and lightest replication method of replicating one DD to another.
Directory replication is more flexible in its ability to configure replicated data from one DD to point to multiple destinations and selecting just the data to be replicated. CIFS and NFS shares can be sent to the same DD destination, but not within the same directory. When directory replication is configured, the destination folder will be automatically created if it does not exist, and after created ownership and permissions on files and folders is maintained. The directory replication process is triggered when a file closes, but to conserve high-demand bandwidth can be forced to a specific time. Snapshots are not replicated automatically and must be configured separately.
MTree replication is a new feature with the introduction of MTrees. In this method, the destination MTree is set read only, and the main difference is in how data is determined for transfer. MTree replication still sends changed data only, but instead of sending vector summaries back and forth, MTree replication creates a snapshot and compares one to another. MTree replication can use encryption and Retention Lock as well.
Pool replication is exactly like directory replication, but the source is VTL data. A separate VTL license is not needed onthe destination DD if VTL restores will not be done directly from it.
DD systems can replicate in the following topologies:
- One to one
- Many to one
- Cascading
- Bi-directional
- ONe to many
- Cascading one to many
Data Domain File System and Data Management
The Data Domain file system starts with the ddvar directory. In NFS it can be shared as /ddvar and in CIFS it can be shared as \ddvar. You can't rename or delete it because it holds the operating environment binaries and configuration.
The data directory is mounted at /data/col1/. "Col1" is short for "collection #1" and there is only one collection at this point. Inside /col1 you will find folders, or "MTrees." The default is "backup," and up until recently all backup data was stored there. With later releases, however, you can create and administer more MTrees under /col1.
Each MTree can be individually managed with its own policies and permissions. The max number of MTrees is 100, however after 14 there is a performance degradation in environments where there is a lot of read/write streams to each MTree. Best practice dictates that you keep MTrees to 14 and to aggregate operations to one MTree where possible. You can mount MTrees like any other CIFS/NFS share, however it is not recommended to mix protocol type. VTL and DDBoost clients create their own MTree, as well.
MTrees are configurable with quotas. Hard quotas stop any data from being written, where soft quotas send warnings and alerts. These quotas can be used in charge-back scenarios, and administrators can be alerted when thresholds are reached to ease administration.
Data Domain Snapshots create a full copy of the data at a point in time (PIT). A DD snapshot copies the pointers from the production data, and then marks new data with a new pointer leaving the old in place for the snapshot. Snapshots are kept in /backup/.snapshot. The max number of snapshots is 750, with warnings at 90% (from 675-749) and alerts at 750.
If the DD is a collection source, snapshots are replicated, but if the DD is a directory replication source, snapshots are not replicated. One other benefit of using snapshots is that there is no additional license required.
Fastcopy is a feature often used to revert from a snapshot. It copies the pointers in full, but where a snap is read-only, a Fastcopy copy is read/write.
Data Sanitization is essentially electronic shredding of data. It is a filesys clean operation, including overwriting free space, metadata and reference data. The sanitization deletes unique pieces of the file only, but if blocks are being used by other files in the deduplicated data, those pieces are left in tact to maintain the integrity of the other files. This feature is often required by DoD/NIST compliance organizations. Sanitization can only be accessed through the command-line, and it is licensed along with the Retention Lock license.
Filesystem Cleaning is a process that runs by default every week on Tuesday at 6:00 AM. When data is deleted, it remains on the disk until the retention lock period expires ur until the cleaning process runs. Space on the DD is not recovered until the filesystem is cleaned. Cleaning self-throttles to 50% CPU and is configurable by an administrator.
To clean the filesystem, data is copied forward to a free container, after which unclaimed data is deleted and finally the original container is deleted.
Monitoring Filesystem Usage depends largely on the rate of change within your data set. If your data changes often and your retention period is long your data on the DD will grow rapidly as well. The size and number of data sets will affect the growth of data on the DD, as well does "compressibility." Essentially, data that is already compressed or deduplicated will not compress well, so your DD data set will actually grow. Modifying the retention period is one way to alleviate this problem. Data can be graphed from 7-120 days to monitor and maintain utilization.
The data directory is mounted at /data/col1/. "Col1" is short for "collection #1" and there is only one collection at this point. Inside /col1 you will find folders, or "MTrees." The default is "backup," and up until recently all backup data was stored there. With later releases, however, you can create and administer more MTrees under /col1.
Each MTree can be individually managed with its own policies and permissions. The max number of MTrees is 100, however after 14 there is a performance degradation in environments where there is a lot of read/write streams to each MTree. Best practice dictates that you keep MTrees to 14 and to aggregate operations to one MTree where possible. You can mount MTrees like any other CIFS/NFS share, however it is not recommended to mix protocol type. VTL and DDBoost clients create their own MTree, as well.
MTrees are configurable with quotas. Hard quotas stop any data from being written, where soft quotas send warnings and alerts. These quotas can be used in charge-back scenarios, and administrators can be alerted when thresholds are reached to ease administration.
Data Domain Snapshots create a full copy of the data at a point in time (PIT). A DD snapshot copies the pointers from the production data, and then marks new data with a new pointer leaving the old in place for the snapshot. Snapshots are kept in /backup/.snapshot. The max number of snapshots is 750, with warnings at 90% (from 675-749) and alerts at 750.
If the DD is a collection source, snapshots are replicated, but if the DD is a directory replication source, snapshots are not replicated. One other benefit of using snapshots is that there is no additional license required.
Fastcopy is a feature often used to revert from a snapshot. It copies the pointers in full, but where a snap is read-only, a Fastcopy copy is read/write.
Data Sanitization is essentially electronic shredding of data. It is a filesys clean operation, including overwriting free space, metadata and reference data. The sanitization deletes unique pieces of the file only, but if blocks are being used by other files in the deduplicated data, those pieces are left in tact to maintain the integrity of the other files. This feature is often required by DoD/NIST compliance organizations. Sanitization can only be accessed through the command-line, and it is licensed along with the Retention Lock license.
Filesystem Cleaning is a process that runs by default every week on Tuesday at 6:00 AM. When data is deleted, it remains on the disk until the retention lock period expires ur until the cleaning process runs. Space on the DD is not recovered until the filesystem is cleaned. Cleaning self-throttles to 50% CPU and is configurable by an administrator.
To clean the filesystem, data is copied forward to a free container, after which unclaimed data is deleted and finally the original container is deleted.
Monitoring Filesystem Usage depends largely on the rate of change within your data set. If your data changes often and your retention period is long your data on the DD will grow rapidly as well. The size and number of data sets will affect the growth of data on the DD, as well does "compressibility." Essentially, data that is already compressed or deduplicated will not compress well, so your DD data set will actually grow. Modifying the retention period is one way to alleviate this problem. Data can be graphed from 7-120 days to monitor and maintain utilization.
Subscribe to:
Posts (Atom)