A few notes on the reporting capability of DPA. DPA offers real-time visibility into the entire backup environment. Alerts can be sent via SMTP or SNMP, and DPA will flagged missed SLAs, backup jobs that run longer than average and on data loss exposure.
The Capacity report trends and forecasts system utilization, such as tape or disk capacity.
The Change Management tracks changes made throughout the entire system and correlates them across the environment.
The Backup Report Card is an at-a-glance report showing backup of every system over a period of time in blocks. Green = good, red = bad. Allows drilling into backup job for details and statistics as well.
The DPA system assists in troubleshooting by correlating data from one end of the backup system to the end of it. It will bring together data from each system in the backup job's life, giving visibility to problem areas for rapid resolution.
Working my way toward being a Full Stack Engineer. I work for a state University leading the Systems Team, looking to chart a future of the data center and how it looks to build into the cloud in a responsible and innovative manner. This blog is largely a place to stash things I pick up in daily work life and pursuit of knowledge.
Wednesday, November 6, 2013
Data Protection Advisor Architecture
DPA is comprised of 3 components as of version 6.0. There is an application server, a data store and an agent. The Application Server and Datastore can be installed on the same box or separated for scalability.
The applications server is the main interface and replaces several services previously in use by DPA such as the controller, the Reporter, the Listener, the publisher and the analysis engine. The data store is a PostgreSQL database and is embedded in the project, replacing the Illuminator, Configuration and Datamine databases. This gives tighter integration and better performance.
The Data Collection Agent gathers data. It is automatically installed on the Datastore and Application Server, and can also collect data remotely using a proxy server for systems for which there is no installable agent, such as data switches and Fibre switches. Lilnux servers require an agent be installed and can not proxy data. The DPA version 6 supports the newer agents as well as DPA 5.5 and newer collectors, and sends the data via http and XML.
DPA uses REST (REpresentational State Transfer) to interface the components. The CLI, the GUI and the agents all communicate with the App server through the REST API. The REST API also communicates with the messaging service and the Datastore. DPA 6.0 uses a high-performance messaging bus, which utilizes an append-only journal that fits in a single disk cylander for performance. REST also launches responses to events in the UI.
DPA is built on JBoss, which is owned by RedHat, and uses PostgreSQL 9.1 as the database. No users are able to access the database without using the "apollosuperuser" account unless they acces sit through the REST API. The apollosuperuser account is a non-privileged account with full access to the database.
Files are kept in "/opt/emc/dpa/services/datastore/data and /engine. The transaction log is pg_xlog, and log rotation happense automatically at 250MB. The max connections to the datastore is 100.
DPA is event-driven, or uses the Event Driven Architecture. The analysis engine uses policies and compares data with those policy conditions to provide recommendations and events.
DPA will run on Solaris 10-11, Windows 2003-2008R2, and RedHat or SuSE linux on 64-bit platforms with 8 GB of memory. If DPA's components are all on one box, 4 CPUs is required, and if they are split out, 2 CPUs per server are required.
The main ports for DPA are 3741, 9002 and 9003. 9002 is the HTTPS connection from agents to server, 9003 is PostreSQL and 3741 is the agent HTTP connection.
DPA Agent will run on Windows, Solaris, Linux, HP-UX and AIX, both 32- and 64-bit architectures.
DPA can be licensed in a couple ways. The DPA for backup is the main application and provides the majority of function. It is licensed by the number of clients unless you are using Avamar, at which point it is licensed by capacity like Avamar to keep things consistent. There is a DPA for Replication Analysis, which is licensed by TB for Symmetrix and RecoverPoint, and per array for VNX systems.
DPA for VMWare is licensed by ESXi host and provides for unlimited VMWare VMs.
VNX File, Celerra and Clariion require no license for use with DPA.
The applications server is the main interface and replaces several services previously in use by DPA such as the controller, the Reporter, the Listener, the publisher and the analysis engine. The data store is a PostgreSQL database and is embedded in the project, replacing the Illuminator, Configuration and Datamine databases. This gives tighter integration and better performance.
The Data Collection Agent gathers data. It is automatically installed on the Datastore and Application Server, and can also collect data remotely using a proxy server for systems for which there is no installable agent, such as data switches and Fibre switches. Lilnux servers require an agent be installed and can not proxy data. The DPA version 6 supports the newer agents as well as DPA 5.5 and newer collectors, and sends the data via http and XML.
DPA uses REST (REpresentational State Transfer) to interface the components. The CLI, the GUI and the agents all communicate with the App server through the REST API. The REST API also communicates with the messaging service and the Datastore. DPA 6.0 uses a high-performance messaging bus, which utilizes an append-only journal that fits in a single disk cylander for performance. REST also launches responses to events in the UI.
DPA is built on JBoss, which is owned by RedHat, and uses PostgreSQL 9.1 as the database. No users are able to access the database without using the "apollosuperuser" account unless they acces sit through the REST API. The apollosuperuser account is a non-privileged account with full access to the database.
Files are kept in "/opt/emc/dpa/services/datastore/data and /engine. The transaction log is pg_xlog, and log rotation happense automatically at 250MB. The max connections to the datastore is 100.
DPA is event-driven, or uses the Event Driven Architecture. The analysis engine uses policies and compares data with those policy conditions to provide recommendations and events.
DPA will run on Solaris 10-11, Windows 2003-2008R2, and RedHat or SuSE linux on 64-bit platforms with 8 GB of memory. If DPA's components are all on one box, 4 CPUs is required, and if they are split out, 2 CPUs per server are required.
The main ports for DPA are 3741, 9002 and 9003. 9002 is the HTTPS connection from agents to server, 9003 is PostreSQL and 3741 is the agent HTTP connection.
DPA Agent will run on Windows, Solaris, Linux, HP-UX and AIX, both 32- and 64-bit architectures.
DPA can be licensed in a couple ways. The DPA for backup is the main application and provides the majority of function. It is licensed by the number of clients unless you are using Avamar, at which point it is licensed by capacity like Avamar to keep things consistent. There is a DPA for Replication Analysis, which is licensed by TB for Symmetrix and RecoverPoint, and per array for VNX systems.
DPA for VMWare is licensed by ESXi host and provides for unlimited VMWare VMs.
VNX File, Celerra and Clariion require no license for use with DPA.
Data Protection Advisor Overview
Data Protection Advisor, or DPA, is a monitoring and reporting tool for backup environments. It is a single tool to manage, report provide trending and analysis of the backup environment end-to-end, from storage to servers to network to applications. It is installable by the customer, and can be done in a couple hours.
Tuesday, November 5, 2013
Managing Avamar Server Capacity
The goal for an Avamar server is to reach a Steady State Capacity. This is the point where backup initialization levels off and the amount of new data brought into the Avamar server is approximately the same as the space being freed by Garbage Collection. Typically, Avamar will reach steady-state shortly after the longest retention period.
The OS View is the total file system in the Avamar, while the gsan View is the total allocated to stripes. Gsan view is 65% of the Avamar server and it is used mainly for user data - this is the licensed capacity. Above that, 20% of the system file space is used for checkpoint overhead and the remaining 15% is for the operating system.
Once the Avamar OS view reaches 65% utilization the Avamar becomes read-only. The amount of primary data on clients, the initial backup commonality, day-over-day commonality and retention policies all affect utilization.
Checkpoint overhead is teh data that changes after a checkpoint is created. When a checkpoint is created, it creates a read-only hard link to the strip. When that data is modified, the maintains the read-only link and creates a new read/write stripe with the changed data. Therefor, the longer checkpoints are created, the more disk is used because there is more read-only data located on the system. Length of time checkpoints are retained is the largest contributing factor in checkpoint overhead, but how empty the stripes are, daily change rate and HFS checks not completing also contribute.
Capacity Threshold Warnings:
80% of user capacity = warning, start planning for expansion or cleanup
95% user capacity is the health check warning, new backups are suspended
100% user capacity makes the Avamar read-only, can still restore from server
85% of OS capacity = Garbage collection stops running, utilization increases rapidly
90% OS capacity = HFS checks stop running
96% OS capacity = no more checkpoints
At 95% health check warning, new backups are suspended and will not start again until the alert is acknowledged.
Monitor with Avamar Administrator or Enterprise Manager. EM has the benefit of predictive analysis, so it can estimate system utilization rates. It also has a graph showing daily change rate, which is a valuable tool in planning.
The DPN Summary in Avamar Administrator shows backup stats, including TotalByts which is all the modified and non-modified bytes being scanned for changes on the client, and ModSent which is the actual bytes sent over the wire.
The OS View is the total file system in the Avamar, while the gsan View is the total allocated to stripes. Gsan view is 65% of the Avamar server and it is used mainly for user data - this is the licensed capacity. Above that, 20% of the system file space is used for checkpoint overhead and the remaining 15% is for the operating system.
Once the Avamar OS view reaches 65% utilization the Avamar becomes read-only. The amount of primary data on clients, the initial backup commonality, day-over-day commonality and retention policies all affect utilization.
Checkpoint overhead is teh data that changes after a checkpoint is created. When a checkpoint is created, it creates a read-only hard link to the strip. When that data is modified, the maintains the read-only link and creates a new read/write stripe with the changed data. Therefor, the longer checkpoints are created, the more disk is used because there is more read-only data located on the system. Length of time checkpoints are retained is the largest contributing factor in checkpoint overhead, but how empty the stripes are, daily change rate and HFS checks not completing also contribute.
Capacity Threshold Warnings:
80% of user capacity = warning, start planning for expansion or cleanup
95% user capacity is the health check warning, new backups are suspended
100% user capacity makes the Avamar read-only, can still restore from server
85% of OS capacity = Garbage collection stops running, utilization increases rapidly
90% OS capacity = HFS checks stop running
96% OS capacity = no more checkpoints
At 95% health check warning, new backups are suspended and will not start again until the alert is acknowledged.
Monitor with Avamar Administrator or Enterprise Manager. EM has the benefit of predictive analysis, so it can estimate system utilization rates. It also has a graph showing daily change rate, which is a valuable tool in planning.
The DPN Summary in Avamar Administrator shows backup stats, including TotalByts which is all the modified and non-modified bytes being scanned for changes on the client, and ModSent which is the actual bytes sent over the wire.
Monday, November 4, 2013
Avamar Server Maintenance Activities
There are server maintenance activities that need to happen daily, which are Garbage Collection (GC), checkpoints and HFS Checking.
Garbage Collection finds orphaned chunks and removes them during the blackout window. If a backup runs long, GC can not start, and if GC has started then a backup cannot start. The server goes into read only mode during GC, so restores can complete. However, if server capacity becomes greater than 85%, any running backups will cancel and GC will automatically run. GC will normally delete any backup jobs that have been deleted, expired or partial backups that are more than 7 days old. During the blackout window, Asynchronous Crunching also takes place, which is essentially a defrag-type job for striped data. It recognizes deleted data and moves other stripes into proximity.
Checkpoints are read only snapshots of the Avamar server, and they enable server rollback in the event of a problem. They are run twice daily, at the beginning and at the end of the maintenance window. Avamar will keep the last 2 checkpoints and at least one validated checkpoint. Checkpoints can be created, modified and executed manually. The older the checkpoint, the more space they will consume.
Validation is when Avamar scans the stripes and validates the integrity using hfscheck. The server is read only when the hfscheck is initiated, but then returns to normal operation following. There is full validation and rolling validation. Full validation scans all data, while rolling validation scans new data and some modified. Validation is the bulk of the maintenance window.
Avamar Administrator is used to monitor maintenance activities under the "Server Details" tab. If maintenance activities are suspended, they will not run until re-enabled by selecting teh maintenance activity and choosing actions -> resume maintenance. **If maintenance activity is not suspended and not running, contact EMC technical support.
Default schedule for windows:
Backup window runs at 8:00 PM and runs for 12 hours until 8:00 AM.
Blackout window runs immediately following the backup window and runs for 3 hours, from 8:00 AM to 11;00 AM. It is mainly running the asynchronous crunching and GC processes. Backup can not be run during the blackout window, but restores can be run.
The maintenance window runs following the blackout window for the remainder of time until 8:00 PM when the backup window begins again. It is primarily running HFS checks at that time, and there are a limited number of backups that can be run. Normally 27 sessions can run, but during maintenance only 3 can be run. It also takes a checkpoint and validates it.
Garbage Collection finds orphaned chunks and removes them during the blackout window. If a backup runs long, GC can not start, and if GC has started then a backup cannot start. The server goes into read only mode during GC, so restores can complete. However, if server capacity becomes greater than 85%, any running backups will cancel and GC will automatically run. GC will normally delete any backup jobs that have been deleted, expired or partial backups that are more than 7 days old. During the blackout window, Asynchronous Crunching also takes place, which is essentially a defrag-type job for striped data. It recognizes deleted data and moves other stripes into proximity.
Checkpoints are read only snapshots of the Avamar server, and they enable server rollback in the event of a problem. They are run twice daily, at the beginning and at the end of the maintenance window. Avamar will keep the last 2 checkpoints and at least one validated checkpoint. Checkpoints can be created, modified and executed manually. The older the checkpoint, the more space they will consume.
Validation is when Avamar scans the stripes and validates the integrity using hfscheck. The server is read only when the hfscheck is initiated, but then returns to normal operation following. There is full validation and rolling validation. Full validation scans all data, while rolling validation scans new data and some modified. Validation is the bulk of the maintenance window.
Avamar Administrator is used to monitor maintenance activities under the "Server Details" tab. If maintenance activities are suspended, they will not run until re-enabled by selecting teh maintenance activity and choosing actions -> resume maintenance. **If maintenance activity is not suspended and not running, contact EMC technical support.
Default schedule for windows:
Backup window runs at 8:00 PM and runs for 12 hours until 8:00 AM.
Blackout window runs immediately following the backup window and runs for 3 hours, from 8:00 AM to 11;00 AM. It is mainly running the asynchronous crunching and GC processes. Backup can not be run during the blackout window, but restores can be run.
The maintenance window runs following the blackout window for the remainder of time until 8:00 PM when the backup window begins again. It is primarily running HFS checks at that time, and there are a limited number of backups that can be run. Normally 27 sessions can run, but during maintenance only 3 can be run. It also takes a checkpoint and validates it.
Avamar Monitoring
Avamar Monitoring is handled through the Avamar Enterprise Manager, the Avamar Administrator and ConnectEMC.
The Enterprise Manager is a monitoring and reporting tool only. No configuration is able to be done, however it is possible to configure one Avamar system to act as the main Enterprise Manager server and point others at it. You can monitor multiple Avamar servers in this way. The URL is http://<avamar_server_or_IP/em.
From the EM you are able to review policy, backup job status, pull reports of performance and capacity as well as predict future capacity required. There is also an update tool that can be installed on a Windows system that will query EMC's online repositry, download patches and stage them for deployment. Clients with avamar client software installed can have the agent pushed from the Enterprise manager.
Avamar Administrator will display 72 hours up to 5000 lines. Its event management will display the last 5000 events from the last 24 hours, and the audit log is available to monitor user actions. THere is also a status indicator for critical services located at the bottom of every view in Avamar Administrator.
ConnectEMC integrates with ESRS. You configure an SMTP server and it will send daily summaries of product info to EMC. This is an optional program. Avamar Administrator will also provide several canned reports, and SQL queries can be run against the PostgreSQL database for custom reporting capability.
The Enterprise Manager is a monitoring and reporting tool only. No configuration is able to be done, however it is possible to configure one Avamar system to act as the main Enterprise Manager server and point others at it. You can monitor multiple Avamar servers in this way. The URL is http://<avamar_server_or_IP/em.
From the EM you are able to review policy, backup job status, pull reports of performance and capacity as well as predict future capacity required. There is also an update tool that can be installed on a Windows system that will query EMC's online repositry, download patches and stage them for deployment. Clients with avamar client software installed can have the agent pushed from the Enterprise manager.
Avamar Administrator will display 72 hours up to 5000 lines. Its event management will display the last 5000 events from the last 24 hours, and the audit log is available to monitor user actions. THere is also a status indicator for critical services located at the bottom of every view in Avamar Administrator.
ConnectEMC integrates with ESRS. You configure an SMTP server and it will send daily summaries of product info to EMC. This is an optional program. Avamar Administrator will also provide several canned reports, and SQL queries can be run against the PostgreSQL database for custom reporting capability.
Avamar Replication
Like Data Domain, Avamar replicates data that is deduplicated, so only new data is sent over the wire. THe tools to perform replication are Avamar Administrator or also the CLI command "replicate."
Standard replication is similar to DD's directory or MTree replication in that specific data is selected and replicated to the secondary Avamar. THis can be in a 1-to-1, many-to-one and bidirectional replication because the Avamar can be both source and destination of replicated data. Full Copy replication is similar to DD's full copy. It is also called "root-to-root" replication and does a complet copy of the source Avamar to the destination. This can only be a one-to-one copy and all data is overwritten on the destination.
Standard replication, when configured, creates a special REPLICATE domain on the destination server. Client backup, server state and config data are all replicated while incomplete backup jobs are not. Restore can take place directly from the destination Avamar without needing to restore to the original Avamar. It handles restore as a redirected restore from a client on the Avamar to a different client.
Full copy replication is typically only used in migrating from one Avamar to a new Avamar. When full copy is used, the Client ID's are migrated also and the clients all appear to be registered with the new Avamar automatically. The clients just need to be redirected to the new Avamar, and this can be accomplished using DNS.
Best practice dictates that a longer time-out window would be used during initialization to accommodate the large amount of data that needs to be transferred. While there are options to "include" and "exclude" clients from replication it is recommended to only use them with extreme caution. Once "include" settings are configured, no new clients will replicate without being specifically added to the list.
Replication is best configured to begin 1-2 hours after the backup commences to get the bulk of backup out of the way, and to end the replication before the blackout window begins. Ensure there is adequate bandwidth to complete the replication within 4 hours of starting - replication will consume 60-80% of available bandwidth.
Standard replication is similar to DD's directory or MTree replication in that specific data is selected and replicated to the secondary Avamar. THis can be in a 1-to-1, many-to-one and bidirectional replication because the Avamar can be both source and destination of replicated data. Full Copy replication is similar to DD's full copy. It is also called "root-to-root" replication and does a complet copy of the source Avamar to the destination. This can only be a one-to-one copy and all data is overwritten on the destination.
Standard replication, when configured, creates a special REPLICATE domain on the destination server. Client backup, server state and config data are all replicated while incomplete backup jobs are not. Restore can take place directly from the destination Avamar without needing to restore to the original Avamar. It handles restore as a redirected restore from a client on the Avamar to a different client.
Full copy replication is typically only used in migrating from one Avamar to a new Avamar. When full copy is used, the Client ID's are migrated also and the clients all appear to be registered with the new Avamar automatically. The clients just need to be redirected to the new Avamar, and this can be accomplished using DNS.
Best practice dictates that a longer time-out window would be used during initialization to accommodate the large amount of data that needs to be transferred. While there are options to "include" and "exclude" clients from replication it is recommended to only use them with extreme caution. Once "include" settings are configured, no new clients will replicate without being specifically added to the list.
Replication is best configured to begin 1-2 hours after the backup commences to get the bulk of backup out of the way, and to end the replication before the blackout window begins. Ensure there is adequate bandwidth to complete the replication within 4 hours of starting - replication will consume 60-80% of available bandwidth.
Subscribe to:
Posts (Atom)