Showing posts with label Avamar. Show all posts
Showing posts with label Avamar. Show all posts

Friday, April 8, 2016

Avamar Commands

Avamar MCCLI commands I use regularly:

View all clients (pipe to grep for selective search):
admin@AVAMAR001:~/>: mccli client show --recursive=true

Retire a client:
admin@AVAMAR001:~/>: mccli client retire --name=</path/to/client>

Tuesday, December 10, 2013

Design Imapact of Avamar System Activities

Avamar runs several system operations that will have an effect on the system's performance.

The backup window is the period of time allocated to running backup jobs, during which no maintenance operations are run.  It is best practice to run all backup to complete in the beginning for the backup window to not risk running into the blackout window.  Use the default window where possible and be aware of time zones when backing up desktops and laptops.

The Blackout Window is a time when activities requiring unrestricted access to the Avamar server take place, such as garbage collection and checkpoint.  During this time, the server is read-only and no backup is allowed.  Use the default window where possible and make sure that there is sufficient time to perform the garbage collection since any change to the window will significantly impact the server's performance.

After the Blackout Window is a Maintenance Window, during which time HFS checks are running.  The server can run a backup at this time, but performance will be slow.  Restores can also be performed.  It's best to keep to the default schedule allowing adequate time for completion.  It's also best to limit ad-hoc maintenance activity and to not run backup during this window.

 Replication should be scheduled to run regularly, typically at the end of the backup window.  Since most backup will finish within 1-2 hours of starting, the rest of the time can be scheduled for replication.  If there is limited throughput for client backup, these can overlap.  The goal is to schedule enough time to allow all daily replication to be completed within 4 hours of starting, allowing peaks of 8 hours, and for high-latency WAN networks (greater than 100 ms), tune TCP buffers on both sides of the connection.

Avoid using the --include parameter when specifying which clients to participate in the replication process, opting for --exclude of particular clients that should not participate.  This makes it less likely to mistakenly omit a system that should be replicated.  Always set the --retention-type option to replicate all retention types, including "none". 

Configure the destination to perform a checkpoint directly after a successful replication so you have a reliable rollback point including replicated data, and increase the time-out period for initial replication period and shorten after it normalizes.

Server performance is impacted as it becomes more full.  Daily maintenance will take longer the more fuller the system is, so it's best to limit storage capacity to 80% and garbage collection is completing, then monitor daily change rate and backup retention to prevent passing the read-only threshold.

The Avamar Server has some important thresholds for capacity:

The gsan process is allocate 65% of the entire system.  Within the gsan view, once you hit 80% utilization you are warned until you hit 95% when new backups are suspended and pop-up alerts are sent when logging on to Administrator.  At 100% utilization, server is read-only and tech support needs to be engaged to assist with reducing usage.

When the OS capacity hits 85%, garbage collection stops running until checkpoints start to roll off at which point it will continue.  HFS Checks stop running at 90% full, and at 96% you no longer get checkpoints, either.

Checkpoints are run twice daily.  When checkpoints are running, no other maintenance activities will begin.  All will be queued up to start once the checkpoint is complete.  As the Avamar becomes more full, checkpoint overhead grows faster and system disk utilization increases because the stripes are more full and get reused.  It's best to keep storage utilization below 80% and that garbage collection completes daily.  Also, leave checkpoint creation and retention to default - keeping last two checkpoints with the last one validated.

HFS Checks validate the checkpoint integrity, and when they are running other maintenance jobs except garbage collection can start.  All backup work orders will queue, and there needs to be 2 or less backups running per storage node for it to begin.  The rolling HFS check is run daily.

Garbage Collection is the single most important process to ensure a consistent state of the Avamar.  If garbage collection is running, the system is in a read-only state and no other maintenance or backup can run.  Coincidentally, if backup is still running, garbage collection will not begin.  Garbage collection can free up about 10GB/node/hour once the Avamar is in a steady state, but this will take longer as utilization of the Avamar grows.

When replicating from a source Avamar server, all other maintenance activities can start and all backup work orders are queued.  When receiving data, garbage collection can not start and all backup work orders are queued.  Be sure to schedule replication and that it completes daily to ensure that there is system redundancy.

Avamar Integration with Sybase ASE

Install the Avamar Client and the Avamar Sybase ASE plugin.

Restore of a database can only take place if it already exists on the destination Sybase server with the same storage layout as the source.  Recovers to current time by restoring last backup plus t-logs from oldest to newest.


Avamar Integration with Hyper-V

Avamar's 6.1 Hyper-V supports Image and Guest backup, including GLR.

For image-level backup, the client is loaded into the Hyper-V management OS, allowing backup of the VMs, and can be installed in a clustered environment by installing the Windows client, then the Hyper-V plug-in, then the Avamar Windows Cluster Configuration Tool.

Restore can take place to the original host, a different host, or machine files for manual provisioning.  GLR is also possible for individual files and folders on the Hyper-V server.

Avamar Integration with Domino

The Avamar plug in for Domino is similar to other plug ins in that the Avamar OS client and the Domino plug-in are required.  Domino clustering is supported, and the Avamar Cluster Plug-in needs to be loaded on the active node.

Hot and cold backup are both supported, as are backup of DAOS-enabled servers and Domino partitioned servers.  Full, incremental and subset backups are supported.

Don't backup during database compaction, and set an exclusion list for temp files.  If t-logs are set to circular or linear, incremental backup will fail.  They need to be set to archive.

When restoring, make sure the user has read/write permission to the directory, and make sure the database is not in use.

Avamar Integration with SAP

The Avamar plug in for SAP with Oracle uses the BR*Tools backint interface to back up and restore the Oracle database.  So on a supported SAP server there will be the SAP Oracle database, the SAP BR*Tools, and Avamar client and the SAP with Oracle plug-in.  After installing, the SAP plug-in needs to be linked to the Avamar server and the backup device type set.  SAP high-availability can be installed on either Solaris or Windows.


The SAP with Oracle plugin supports the same backup types for the database as the Oracle plug-in, and can be scheduled or run on-demand.


When a backup request is sent from the mcs process on the Avamar server, the avagent spawns the avsap process, which then runs the brbackup (or brarchive), which runs the backint process, which then uses the avtar process to send data to the gsan process.  When the backup is run from the cli, the administrator is going to directly run the brbackup, which spawns the backint and one or more avtar processes.

REstore can only be done through the CLI, not through the Avamar Administrator.  Up to 10 sessions are supported in a multi-streaming restore.  In a restore, the backint process is actually run from the brrestore or brrecover, which queries the Avamar server for the backup files and starts the restore using avtar.


Avamar Integration with VMWare

There are two ways to backup VMWare using Avamar.  One is to perform a regular guest backup, installing an agent as if the VM were a physical machine.  The other is using VMWare's vStorage API for Data Protection, or VADP.  This uses a proxy node and has an automatic detection of virtual machine clients within the vCenter and is an efficient method of backing up entire images.  It functions for Windows and Linux VMs, including selective restore of individual files and folders.  The proxy is a Linux VM running Avamar software, deployed using an OVA.

Guest backup gives the highest level of dedupe and allows applications to be gracefully quiesced.  Any OS that is supported by Avamar is supported and all normal limitations apply regarding power and connectivity.  Restoring from guest VM backup typically employs a "bootstrap" image that is installed with applications and OS followed with a restore of unique data.

Image backup is performed through a proxy and is integrated with Avamar Administrator.  It tracks changed blocks (CBT) so they don't have to be scanned each backup, saving considerable amounts of I/O.

Image backup creates a snapshot and backs up the snapshot using a proxy to handle the deduplication.  Multiple proxies can be used for load-balancing.  You need a different proxy for Windows and Linux, and file-level restore is supported up to 5000 files.  Images can be restored to the original or a new VM.

vCenter groups are created in the Avamar Administrator in the vCenter domain and is where you specify which proxy is used.  The default VM group cannot be deleted.  There is also a default proxy group that holds all proxies and cannot be deleted.

Image level restore can be  to the original VM, a different VM or a new VM.  The target VM must be powered off. 

File level restores into the VM File System but ACLs are not restored.  Encrypted and redirected files must be restored to the original location, and the client must be powered on.

Using FQDN for ESX servers will reduce network lookup traffic for vCenter groups, and it is also a good idea to throttle guest backup on the ESX host so as not to consume all available resources.


AVamar Integration with NDMP

The Avamar NDMP accelerator supports VNX, Celerra, and NAS devices running Data ONTAP.  It supports NDMP version 4 over 100/1000 Mb/sec Ethernet and will work over a LAN or WAN.

A single NDMP accelerator appliance will support multiple storage devices and up to 8 simultaneous streams, performing real-time data dedupe.  Of course, there needs to be sufficient RAM and configuration on the accelerator.

 By default, NDMP backups contain all storage device accounting information and ACLs.  Only volumes are visible , and include/exclude lists are not supported.  When combined with VNX or Celerra, the NDMP accelerator performs incremental backup at the volume level only and defaults to full backup of sub-root directories.  Best practice is to not backup more than 10 million files in a single backup job, which can be overridden in the config but jobs may fail for lack of memory with too many files.

When backing up NetApp Filers, the NDMP accelerator is backward compatible to Avamar 4.1 servers, but version 5.0 or newer is recommended.  The appliance will backup and restore volumes, qtrees and directories, and SnapVault snapshots must be backed up in full. 

The NDMP accelerator will perform file level restores, as well as single directories and sub-directories, but data backed up on other Avamar client types are not restorable to a NAS - if it came from NDMP, it stays on NDMP and vice-versa.  There is no web restore with external authentication - users must have local authentication, and ACLs are fully supported, but there is no control for overwrite options.

Both LAN and WAN location of the NDMP device are supported, but it is recommended to have the accelerator local to the device being protected.   8 GB of memory is required in the appliance for multiple streams, and the recommendation is to have 4GB of memory per stream (8 streams max).

Multi-volume accounts will attempt to backup all volumes simultaneously, opening up to the possibility that there would be too many simultaneous streams.  It is best practice to combine all multi-volume accounts into one account and have a single multi-volume backup using the avsetupndmp script.


Avamar Integration with DB2

Avamar 6.1 supports DB2 on Windows, AIX and Red Hat or SuSE Linux.

The Avamar client for the OS is required along with the database plugin.  Online, offline and DB2 Partition Backups (DPF) are all supported backup types, and restores from both online and offline backups supported by DB2 are supported by Avamar.  DB2 databases can be backed up and restored only in offline mode, however.

This was the shortest module in the entire session.  If you use DB2, you'll probably put more time and energy into researching this because you'll have more time on your hands, I guess...

Avamar Integration with Oracle

Databases are  backed up in a number of ways, and Avamar will accommodate all of these methods:  Hot backup, cold backup, database dumps, backup/BCV.

Hot or online backup can be done while keeping the database online and in ARCHIVELOG mode.  The tablespaces are set to backupmode and the files are backed up.  These are less disruptive to the environment, and are fast and efficient using RMAN through the command line or Avamar GUI.  However, the control files and archived redo logs are stored separately and there are some additional steps required to configure RMAN.

Cold or offline backup take the database offline, backing up all data log and control files.  This is an easier method to restore from because the database is in a consistent state when backed up.  The database can be in ARCHIVELOG or NOARCHIVELOG mode, but the database is unavailable during backup.

Snapshot backup backs up the BCV (business continuance volume) or snaps.  This is the fastest restore, since the backup copy is immediately available on local disk mirror.  There are no additional host CPU resources because it is done on a proxy host, and non-supported databases with export capability are supported and there is a standard file system backup.  However, additional primary storage is required to support the split-mirror, and an additional server for the proxy is required as well.

Avamar 6.1 client supports Oracle 11g stand-alone configurations on just about every OS Oracle runs on (Windows, UNIX, Linux) and Oracle 11g HA configurations on Windows, Solaris and Oracle Real Application Clusters (RAC). 

Avamar's 6.1 client also supports Automated Storage Management databases, and backup can be taken to and restored to multiple targets.  Up to 6 channels can be allocated for Oracle backup and restore in parallel.  The Oracle plugin will support offline backup, recovery catalog, RAC configs, archive logs as well as Flashback Database Recovery and Corrupt Block Recovery features.

Oracle's RMAN (Recovery Manager) is used to backup a database, tablespace or data files.  RMAN is used to initiate backup and recovery commands from the Avamar plug-in and routes them to the Avamar server.

Avamar supports full backup of all data files and archive logs, Level 0, which is similar to full but is required before using a level 1, and Level 1 backup, which backs up all database blocks changed since the last L0.  This uses BCT (Block Change Tracking) and will force a L0 if none exists, so you may want to create two backup schedules for L0 and L1 backups.

Monday, December 9, 2013

Avamar Integration with Sharepoint

Once again, most versions of SharePoint are supported on most server platforms.  SharePoint 2013 on Windows 2012, 2008R2 SP1 down to SharePoint 2007 SP2/3 on Windows 2008 SP2.  Domain privileges are required for backing up SharePoint, and backup can be done to either Avamar or Data Domain on the back end.

Avamar does federated farm backup and recovery in that it discovers the Sharepoint farm topology and backups up data from appropriate hosts, and will automatically recover to appropriate hosts without having to do individual restore to specific components. It will support up to six parallel backup streams if the host has multiple core CPUs, and can do GLR of Sharepoint pieces using 3rd-party tools such as Kroll's Ontrack.

In a standalone sharepoint farm, all that is required is the Avamar client for Windows and the Avamar plug-in for Sharepoint VSS.  To add in the optional GLR, an additional Windows server or workstation running Ontrack Powercontrols for Sharepoint needs to be added for the GUI to restore individual items.  On the standalone Sharepoint farm, we need the client for Windows. the Sharepoint VSS plug-in back-end option and then also the GLR option.  The Ontrack PowerControls Agent for Content Transfer Service (ACTS) also needs to be installed on the Sharepoint farm.

In a distributed Sharepoint farm, Avamar Windows client gets installed on all systems (including the Ontrack Powercontrols system for GLR if in use).  The Sharepoint VSS plug in gets installed in two components, front-end and back-end.  The front-end goes on the web front-end servers and any other WFE servers not containing sharepoint data.  The Sharepoing VSS back-end plug in gets installed on the search server, the database sever, and the central administrator.  The Ontrack Powercontrols ACTS gets installed on the central administrator, also.

Avamar backs up the entire Sharepoint farm.  This is called a Federated backup.

Avamar will do a full DR restore, it will restore the entire farm or individual components to the original or to alternate locations, and the GLR will restore everything from the web app down to the item level to the original farm, an alternate farm or as local files and folders.

Limitations of the Sharepoint VSS plugin are that individual websites, list or list item is handled at the parent site collection level.  Also, the configuration and central administration databases can only be restored as part of a full farm recovery, although individual "non-config" databases may be restored without full recovery.  The Sharepoint VSS plugin does not allow backup to tape (extended retention) nor does it backup the IIS metabase, so all settings should be well documented for each front-end web servers.  Lastly, only SQL full backups are performed and log truncation will not take place.  Set databases to "simple recovery mode" as a workaround.

Avamar and SQL Server Integration

Most SQL server versions are supported, but of course EMC recommends checking the compatibility list.  Of note is that SQL2012 is supported on Windows 2012 (Exchange 2010 on Windows 2012 is not supported at the time of this post).  As with any Windows application, the Avamar Client for Windows is required.  Also required is the Avamar Plugin for SQL Server, which used SQL VDI to perform backup.

Avamar supports SQL failover clustering by installing Windows agents and SQL plugins on both servers, then running the Avamar Cluster Client on the active node.  When installing on SQL AlwaysOn Availability Groups, the Cluster Client is optional because Avamar will pull data from either node.

The SQL restore will do full, differential and transaction-log (incremental) backup, as well as multi-stream backup.  It will restore an instance, database of filegroup to the original location, to a different instance on the same server, or to an instance on a different server.  You can restore the original database with a new name to the original server, or you can restore the database that is lost or corrupt with the transaction log intact.

Options to control the restore behavior would be to use tail-log backup, point-in-time restore or SQL Server recovery operations.

Avamar and Exchange Integration

Avamar supports Exchange 2007 and 2010 on x64 architecture.  2010SP2 requires 2008SP2/2008R2 and 2007SP3 requires 2003SP2/2008SP2/2008R2.  Exchange 2010 DAG configurations are supported, as are Single-copy Clusters (SCC), LCR, CCR, SCR.

There are 3 Avamar client components for Exchange.  They are Windows client (required for all installations), Avamar Plugin for Exchange VSS (required) and Avamar Plugin for Exchange Granular Level Restore (GLR) which is optional.  Also required is an AvamarBackupUser, and this user must be set up and the Avamar Services set to run under its context.

Exchange 2010 with Exchange VSS plugin is good for database, stand alone (non-DAG) databases, and will backup both active and passive databases in a DAG environment.  Exchange 2007 with the Exchange VSS will backup Storage Groups on active and passive nodes in a CCR/SCR as well as stand along (non-CCR) Exchange servers. 

All Exchange backup requires a Windows client and can be backed up to either Avamar or Data Domain storage. 

The Exchange Cluster Configuration (ECC) Tool is used to configure the cluster on the active node of the Exchange cluster, then settings are replicated to the passive node.  Then use the Avamar Cluster Configuration to configure the backup to run against the passive database to minimize load on active database and on users.

In a 2010 DAG, install the Windows and Exchange VSS on each server in the DAG, then another DAG resource using the ECC Tool.  The DAG client coordinates a federated backup of the passive databases.

The Exchange VSS will do full database DR recovery to the original or new location.  The Exchange GLR will restore to the 2010 RDB or 2007 RSG on either the original or different servers.  GLR is needed for message-level recovery.

Installing GLR requires a reboot of the Exchange server, which can be post-phoned until the moment of restore.  When installing the GLR, while not necessary, it is recommended to move the active databases off the server being installed.  Always use the Avamar Configuration Checker for Microsoft Windows, as well.

Avamar Integration with Data Domain

Benefits of integrating an Avamar system with Data Domain on the backend would be that there is a flexible architecture and lots of plugins for applications.  VMWare integration also takes place with changed-block tracking, file-level restore and Exchange GRT.  It is best utilized in environments where there are large, active databases of greater than 5TB or a high change rate.  There is a maximum of 1000 files per client.

Clients will use DD Boost, so there is no Avamar deduplication and no client cache files to tune.  Avamar runs a process called ddrmaint, which runs on the Avamar utility node for maintenance and garbage collection.  Also, there is no in-flight encryption options with this configuration.

Avamar/DD will support full, incremental, differential and VMWare Changed-block tracking backup.  The full backup must be located on the DD with the incrementals/differentials.  All data goes to the DD except Exchange VSS log files, Exchange VSS backups less than 10MB in size, and SQL-Server tail log backup during a restore.

The Avamar will control the maintenance on the DD as well.  The HFS check will run, but check for the existence of the file and not the file contents.  When backup expires, it moves to teh DELETED folder, and eventually is physically removed when the DD clean operation is run.  This will delete checkpoints and backup that fail as well.  Data shredding is not available in this configuration.

Checkpoint creation copies data from the "cur" directory to "cp<timestamp>", and deleting them moves them to the DELETED directory.  Rollback moves data from "cur" to "cur.<timestamp>", clones cp.<timestamp> to cur and needs to succeed on both the Avamar and DD or the process fails.

Replication is controlled by Avamar, and there needs to be an Avamar and DD at both sites.  The Avamar will replicate itself to the remote Avamar, and the DD will replicate to the remote DD.  In this scenario, we can use a 1-1, one-many, many-one for the data.

To configure the integration, add the Data Domain to the Avamar system and modify the backup policy to use DD as the back end.  This is selected as part of the backup policy (at the data set level).

Avamar 6.1 requires a minimum of DDOS5.0, and licensing for DD Boost and DD Replicator need to be purchased.  DD recommends using the FQDN of the Avamar, and SNMP must be enabled.  Avamar must be configured with the SNMP community string and a read/write user account for it.  Avamar needs to be licensed to include the entire DD capacity, also.

Avamar Tuning - Client Cache Files

Avamar uses two files for tuning performance on the client side.  The File Cache (f_cache.dat) and the Hash Cache (p_cache.dat).  The values affecting these files can be modified with switches and configuration to increase efficiency of the application.

The file cache, or f_cache.dat, is a list of files on the client.  When this file gets a "hit" the Avamar client skips it because it is already on the backup server and a marker is sent in its place.After the initial backup, 98% of the files on the client will be a hit and not sent over the wire.

The hash cache, or p_cache.dat, is a list of the individual chunks of data that have been transferred.  This is most beneficial on database files and large files.

If the file cache is undersized, each file  on the client needs to be opened, read, chunked, hashed and checked for on the server.  This increases processing and memory utilization as well as network to retrieve the cache.  If the hash cache is undersized, it needs to check back with the Avamar server for every chunk.

f_cache.dat is limited to 1/8 physical RAM, and p_cache.dat is limited to 1/16 physical RAM.  The file size will double until it reaches the max instead of incrementing proportionately.

To tune the file cache, the formula is:
(# of million files) * 44 MB

To tune the hash cache, the formula is:
1 MB per 1 GB of database file

Also, it takes about 60 minutes for every million files or 100 GB of database data.  If your backup takes 30% longer than these values, the cache files should be examined for correct sizing.  The avtar process logs the size of the cache files when it starts and also when it ends.

To determine the maximum size for f_cache.dat, use N * 44 MB * 2 and use the "--filecachemax=<size in MB> | <size in fraction>" with the avtar command.  A negative number denotes a fraction (-8 is 1/8, -16 is 1/16).  If size of the f_cache is being reduced, the original must be deleted and restarted.  Also, setting to a number will limit the cache file to the closest "doubling" of the original cache file instead of allowing growth to that max.  Furthermore, the cache file is limited to the last doubling when a multiple is used in the filecachemax number is set instead of using a fraction.

For tuning the p_cache.dat, the max file size is calculated as 2*Y MB, where Y is the number of GB in the data base.  The caveats for f_cache tuning apply also to p_cache.

When tuning the file and hash caches, the goal is to not allow data to go to swap on the hard drive.  Also, do not use more than 25% of physical RAM, and whenever possible use the fraction in filecachemax instead of a specific number.  On a file server, increase the f_cache, although at some point more RAM may simply need to be added when the cache grows larger than 1/8 physical ram.

Another option for tuning cache files would be to have multiple, smaller data sets on a given client.  Then set the "--cacheprefix" to a different value for each data set, which would create a separate cache file for each.  Be sure you have enough disk space to accommodate.  You can also set a different var directory by using the "avtar --vardir=<path>" command.

Saturday, December 7, 2013

Avamar Configuration Design Best Practices

When configuring Avamar clients, the first place to begin is to check all documentation to ensure that all OS and applications are supported.  Secondly, make sure the RTO is achievable and realistic.  If it is neither, consider locating an Avamar server local to the client.

Keep your Avamar system as simple as possible.  Limit domains to situations where administrative duties must be divided, as well as groups and policies.  Policies should be planned well in advance with a simple solution fitting today's need being the best route to take.  Also remember to leave the default group disabled.

It's best practice to use a single data set for Windows, Solaris, Linux, etc.  The data will be skipped if it doesn't match, and it will keep policies to a minimum.  Retention policies should always be at least 14 days, and implement advance retention policies with daily scheduled backup.  Make sure your backup jobs all complete within the backup window and not during blackout or maintenance windows, and after the initial backup be sure to set the expectations with customers and monitor backup for anomalies.  Backup may need to be adjusted if daily change rates are not what were expected, or other factors may be causing differences. 

There are recommended limitations on Avamar clients, and if outside these it may be best to find alternate solutions:
  • 5-10 million files per Avamar client
  • 0.5-2 TB of database data per Avamar client
  • 2-5 TB of file data per Avamar client
When backing up remote clients, RTO is the primary consideration.  Be sure to understand what kind of WAN connectivity exists and how it's being used already before introducing backup traffic.  It's best practice to keep Exchange on the local segment where the Avamar is located, and always be aware of what is being backed up to where.  It may be best to use a centralized Avamar server where administrative resources don't exist at a remote location, or if large multi-node Avamar servers are required at a remote location the cost may be prohibitive.  Best practice states that we use a centralized server unless RTO can not be met.

Keep clients within the same time zone on the same Avamar server, and be cautious of crossing regions so as not to interfere with blackout and maintenance windows.  This principle goes for replication as well.  EMC technical support can modify those windows in cases where there is a requirement.

When backing up desktops and laptops, exclude common data such as OS and application files.  Using the flag #USERDOCS# for Windows systems will backup up the user folder form multiple Windows versions.  Schedule DLO for a time when you know all systems will be online but under-utilized, such as lunch time, and minimize the include/exclude lists for efficiency.

Training users about the backup is also important, and they should know how and why to perform an on-demand backup, and set the Avamar to reject backup requests during the maintenance window to keep things running smoothly.  Multiple backup schedules can also be configured as selectable by the end users to determine which is best for their need, and reminders can be sent along with the ability to suspend or pause them if needed.

For initial backup of Avamar clients, it's best to start with a small number of clients to produce a dataset from which commonality will be generated.  This will increase the backup speed of following clients.  On a new Avamar system, consider enabling overtime because all the data is new and there won't be much maintenance to take place.  Never enable overtime on an existing Avamar, rather configure the use of partial backups to be saved.  Partial backups are saved for 7 days and are not deleted by the garbage collection process.  When an unfinished job ends before completing and partial backups are enabled, the backup job will continue from the last known state and will complete faster.

For a large number of clients, start in small groups.  It's recommended to start at ten times the number of Avamar Storage Nodes, and then double it each day as long as the backup is completing within the backup window.  Reduce the number of clients if the backup window is not being met.  Also consider using the Avamar Client Manager to roll out clients.

When initializing remote clients, the main issue to be concerned about is network outage.  It may be best to seed backup over the LAN or to a USB drive.  To seed to USB, the data is copied to removable media and then attached to an Avamar client local to the Avamar server.  A backup is performed on that local client, and once the data is introduced into the system it will not be duplicated over the WAN.

When initializing clients, large backup may time out at the end of the schedule or at the end of the backup window.  To avoid this, configure partial backup to be kept, and run backup daily - at least to begin.  This way the backup will rapidly complete after which a desired schedule will have commonality and complete faster.  Also run multiple, shorter backup and disable overtime to get partial backup.  If the initial backup is taking too long, check to make sure the cache files are sized appropriately or seed the backup.

The avtar process requires 20-30 MB of memory depending on OS, and client cache files are loaded into memory and grow to a configured max.  The file cache (f_cache) will grow to 1/8 of physical RAM, and the hash cache (p_cache) will grow to 1/16 physical RAM.  The size of the cache files is recorded at start and end of the avtar log for monitoring and tuning assistance.

Avamar Performance Consderations

The typical Avamar life cycle tells us that the initial deduplication rate for non-structured (file system) data is going to be about 70%, while structured data (databases) is going to be about 35%.  There is little garbage collection and the data is mostly unique.

Once we hit the retention period, the Avamar goes into a "steady state" where the amount of new, unique data coming in is the same or less than the amount of data being aged off.  For file data, daily change rate averages 0.3% and database daily change rate averages 3-5%.  Once steady-state is achieved, garbage collection will clean up about 10GB per node, per day.

Ingest performance can be negatively impacted if the nodes don't meet the minimum sequential read, write and random read seek performance requirements.  A less-utilized Avamar server will also perform up to 50% faster than a grid reaching capacity.  A short backup of 1-2 GB new data will backup slightly faster than large backups, and a larger chunk size will also be faster.  As with everything Avamar, doing anything during the maintenance window will decrease performance significantly.

Backup performance has several factors to consider as well, starting with inappropriately sized file and hash caches (tuning of caches will come in a later post).  A client requires sufficient CPU and disk I/O resources, and x86 processors will typically backup faster than SPARC or PPC due to the fact that x86 are typically faster CPUs with less cores than the others. 

Backing up multiple VMs on the same ESX host could cause a performance hit, and it's better to spread VM backup across several hosts if possible.  Network performance and configuration will also come into play, so make sure to match speed and duplex and measure WAN latency.  Similarly, a 100MB connection may be adequate for Avamar backup after initial backup, but may not perform well enough for the initial backup.  Seed systems when possible over more robust connections.

Restore actions will need to be read from the Avamar server, sent to the client and ordered in client memory, uncompressed and finally written to disk.  Therefore, read rate from the Avamar server, network performance and write rate on the client will all be factors.  Use the Avamar Validate function, which is a utility that performs a test restore without writing any data.

When restoring data, several small files will restore slower than a few large files, and again - if the maintenance functions are running the restore will be impacted by up to 70%.  Restoring from multiple nodes of the Avamar grid will be considerably faster than if restoring through a single node, and you can use the "all nodes" option in avtar to set this as the restore method.  The 2nd restore will also go faster than the first because there is data in cache from which we can obtain commonality.

Client restores will have similar limitations, in that file size, number of files, client resources, network and encryption will all be factors.

When replicating data, we're essentially doing an "all node" restore from the data nodes of the source to the utility node of the destination, then backing up the source utility node to the replication target.  WAN can be tuned to 60-80% of available bandwidth, and is almost always the issue with replication.  Start there.

Friday, December 6, 2013

Avamar System Sizing

Industry standard is to report capacity in decimal numbers, whereas Avamar OS reports in binary.  Thus, the system reports 1.8TB on a 2TB system.  Also, RAIN capacity is factored into the capacity report beforehand.

When planning for Avamar capacity, it's important to understand that there are different "views."  The OS View is the total amount of disk space.  Then there is a GSAN view, which is total allocated to stripes minus space freed by garbage collection plus RAIN parity.  The user capacity is 65% of GSAN view, and when user capacity hits 100% it is the read-only threshold.  There is also the CUR view, which is all the current working directories.  In the OS view, 20-25% is reserved for checkpoints, and 15% of the disk is reserved for the OS itself.

Major factors when planning capacity required would be the amount of primary storage (or how much are we backing up?), commonality which increases as retention period increases.  Also to take into consideration is the daily change rate.  Remember to plan for 80% of the user capacity to follow best practice.

The number of nodes should be kept to 12-14 for the initial implementation to allow for growth to the max 16 nodes in the future.  Capacity can not be added to a single node server, either.  A rot-root migration needs to take place to increase a single node, so plan for growth.  The number of client connections are also a consideration in that an Avamar can only handle 27 at a time.  In environments where there are a large number of clients it may be best suited to have several smaller nodes as opposed to one larger node.

Type of data to be backed up plays a part in the sizing requirements, also.  File System, database, rich media and virtual machine files all have different dedupe ratios, and a high daily change rate and lots of files (millions of smaller files) will also increase backup time and disk utilization.

RAIN is random array of independent nodes and provides for failover and fault tolerance across nodes in the Avamar grid.  The minimum number of nodes is 1x3 (1 controller and 3 storage nodes) plus a spare storage node, unless you have premium hardware support at which time the spare is no longer required after GEN4S.

Checkpoints require 20% of the OS view, and are snapshots of the system allowing rollback in case of corruption or operational difficulty. 

Encryption will add 33% to resources, which equates to 33% of the user capacity, so the system will now handle 43% of the user capacity instead of 65%.

The longer retention period you are able to implement, the better your dedupe will be.  Weekly backups have the same amount of unique data as 3 daily backups, and monthly backups have the same amount of data as 6 daily.  Therefor, using a regular daily/weekly/monthly rotation will decrease the amount of disk utilized.  For instance, 30 daily + 2 monthly will result in the initial backup + 42 days of unique data compared to 90 daily backup.  The minimum retention period should be longer than the time between backup so that you don't have to constantly do initial backup because the data aged off.

When clients are distributed geographically, it may make sense from an RTO perspective to have several, smaller nodes closer to the clients in each site rather than centralized nodes for all as long as there is adequate bandwidth and system resources to handle them.

Best practice is to replicate all Avamar servers to another Avamar server.  When a single node is implemented, replication is required - such as in the case of the AVE.  Be sure to size appropriately to accommodate the replication type and amount.

Be sure to consider what types of data are being backed up when AVE sizing.  Will this be file system, Apps (like Exchange, SQL) or mixed?  The daily change rate needs to have an appropriate amount of I/O performance, and when initially implemented the read-only threshold should be set to 40% until actual system metrics can be gathered.  The max daily change rates for AVE are <2GB/day with a total of 650 GB total protected data on a 0.5 TB AVE if backing up non-structured data like file servers, and <5GB/day and 500 GB total protected if structured or mixed data.  The 1 TB unit can handle <4GB/day and 1.3 total protected for non-structured, and <10GB/day and 1 TB of structured data.  The 2TB unity can handle <8GB/day and up to 2.6 TB protected non-structured data and <20 GB/day and 2TB structured.  So the difference between them is that you can have more daily change in a mixed backup environment, but the total protected data reduces.

PAT is a 24-hour test to ensure both acceptable I/O, and that the Avamar server will not only have sufficient resources to run but that it will not have a negative impact on other VMs on the ESX server.  Performance of AVE increases as t6he I/O increases, as well as with a lower daily change rate and less capacity of the AVE used.  When putting in an AVE, be sure to thick-provision and eager-zero the disks, which should be R1 or R10 physically.

Avamar Theory of Operations

There are essentially 3 components to an Avamar system, or grid.  They are the Server, the Administration Console and the Client.

An Avamar server can handle 27 simultaneous client connections, with 1 always being reserved for restore.  A server is comprised of the Utility Node, which has the mcs, ems and cron jobs running.  It listens on port 8443 for web browser requests, 7778 for the admin  console, and communicates with the avagent on port 28001 (the avagent listens for mcs requests on 28002, coincidentally - they are a pair).  The Storage node runs the gsan process and talks to the avtar process on the client over ports 27000 and 29000 for encrypted communication.

The mcs handles notification and reporting for the GUI, SMTP, syslog and SNMP services, while the MCDB is the database that allows communication for reporting, such as with DPA.

An Avamar client runs avagent which listens for backup requests on port 28002 from the Avamar server and spawns avtar, which then sends data to the Avamar storage node (gsan) specified by mcs.

The Avamar server node components are:
  • Utility node - 1U server with no storage
  • m600- 2 TB
  • m1200 - 3 TB
  • m2400/s2400 7.8 TB
  • accelerator and media access nodes
Backup and Recovery Manager is a relatively new product that allows monitoring only and is in development.  It ties in seamlessly with Avamar.

Integrating Avamar with Data Domain is most beneficial in environments where large, active databases are in use.  Data is held on the DD while meta data is on the Avamar.  Avamar will run maintenance routines such as garbage collection, checkpoints, rollbacks, HFS checks, all of which take place on the DD storage.

Integrating Avamar with Networker makes Avamar a deduplication node, mainly, where deduped data is stored on the Avamar, while meta data and deduplication info is stored on the Networker Storage Node.

Integrating Avamar with DPA is relatively straight-forward in that DPA will collect info on the backup system in general.

It is recommended to always configure the Email Home Notification, which sends a report twice daily to EMC with status and errors, events and warnings on the system.  ConnectEMC is another notification tool that ties into ESRS and allows EMC support to be alerted immediately and connect to resolve issues without interaction.

Security is administered in several ways on Avamar.  The first being user roles.  There are Administrator, Operator and User roles.  The administrator has full root access to the system, while operators have access to role-defined privileges.  Operators will typically used to backup a single domain or system.  Users are typically allowed to backup and restore from a single client.

Authentication can be managed locally from the AVS process, Active Directory, NIS, and LDAP.  LDAP groups must be mapped to each domain and role, and they are available for the Administrator, Enterprise Manager and Client Manager but not for local system administration.  This needs to be configured in the Avamar server and usernames and passwords are managed then through the authentication service and no user accounts are local to the Avamar.

Client/Server authentication can be handled by TLS or PKI.  It is best to use a commercial CA if you don't have one yourself, as self-signed certificates are not recommended for production use.  1-way authentication would be where the client requests a certificate from teh server, and 2-way authentication is when the client requests first, and then the server requests authentication from the client.

Web browser security is handled using TLS and SSL.  To reduce errors on the browser, server authentication should be configured where you create a private key and then generate a certificate signing request for it.

Data can be encrypted both at rest on the Avamar as well as in flight from client to Avamar.  The at-rest encryption algorhithm is blowfish, and this is a one-time decision made when the system is installed.  There is no method for removing encryption outside of migration of data.  Data in-flight can be configured on each job, however, and can be changed on the fly.  In-flight encryption is not supported with DD on the backend, however.  The options for in-flight encryption are high, medium and off.

Secure delete is a feature to completely remove data with no chance of retrieving it forensically.  This is done using the "securedelete" command and it performs a multiple write over the disk.  It can only be used on a per-client basis, and any replicated copies of the client need to be manually removed as well.

After Avamar 5.0, address translation, or NAT is supported so that utility nodes can present an IP address on the same subnet as a remote client and the communication will happen automatically.

Avamar Extended Retention gives the ability to take data off the Avamar in non-deduplicated form and written to tape or external media.  This function would need a media access node to complete.

Avamar Virtual Edition is an Avamar VM that runs on n ESX server.  It is a single-node Avamar system, and therefore replication is required.  There are capacity limits of 0.5, 1, or 2 TB making it best used in small and remote environments.  The ESX server must pass a benchmark test called the AVE Performance Assurance Tool (PAT).  AVE can not be expanded.

Operational Best Practices would suggest that the design should always maximize operational availability.  In other words, schedule backup to take place during backup windows and not during maintenance windows.  You should be monitoring the infrastructure as well as the capacity to ensure that there is always ability to run both maintenance jobs as well as network, UPS and cooling capability.  Capacity should be closely monitored to avoid hitting the read-only threshold of the Avamar, and the Avamar requires constant care and feeding to continuously be tuning performance as jobs are added and removed.







Thursday, November 7, 2013

Networker Integration

Networker integrates with VMWare much like Avamar.  It will do guest-based backup with agents installed in the guest VMs for which the VM needs to be powered on.  It will also do image-level backup using VADP using a proxy client that takes a snapshot, mounts the snap and backs up the snapshot.  This reduces overhead on the ESXi host.  Also, like with Avamar, CBT can be used to backup only changed blocks of data in the VMDK files.  Individual file restore from VMDK can take place only on NTFS volumes.

Networker integrates with cloud services, both private and public.  It will compress and encrypt the data in transit, as well as throttle bandwidth to ensure service level to applications and customers.  It will handle backup, recovery, staging and cloning.  Clones are infinitely appendable when storing in the cloud.

Networker integrates with DPA, providing expanded monitoring and predictive analysis.

Networker integrates with Avamar, making Avamar to be seen as a storage device.  It will then  use Avamar's source-based deduplication and the Avamar is referred to as the dedupe node.  One advantage over Avamar is that Networker will allow multiple simultaneous backup of the smae client, for instance you can run a file level and database backup at the same time reducing time.  All jobs are managed and configured through Networker, and all Networker Modules can be used.  One caveat is that Networker will not manage Avamar Replication.  That still needs to be configured and managed through the Avamare interface.

When a backup request is sent from Networker to Avamar, the save command spawns the avasm, which in turn spawns the nsravtar.  The process then follows the normal Avamar backup routine where it checks the hash file on the storage node, creates a hash cache and compares to new data, etc.

Data Domain can act as an AFTD or FTD, it can be utilized as VTL over Fibre Channel, and it can be used wiht DD Boost.  With DD Boost, you oobtain the benefit of central config and management of backup.  Replication is also controlled in Networker when integrated with Data Domain.  You can install DD Boost on a storage noe, or install it on the client and do client direct backup.