Showing posts with label NetWorker. Show all posts
Showing posts with label NetWorker. Show all posts

Tuesday, December 10, 2013

Networker Application Integration

VMWare Integration:
Can do guest or Image-based backup (VADP).  When doing image backups, there is no need for LAN connectivity to the guests and is well suited to DR.  File-level backup/recover is supported on Windows only.  There is little impact on the host and guest.

When defining backup policies for guest-level backup, create backup groups with VMs across multiple ESX server to reduce load on individual hosts due to concurrent backup.  Reduce full backup frequency to decrease I/O and network traffic and use modules for apps and database.

When using VADP, backup processing is offloaded to a proxy and can be done any time if done on another host.  There is file-level recovery only for NTFS, and image recovery of a full VM from an incremental CBT is not possible.

Image-level backup requires a full backup after hardware change, OS patch or update, or driver updates.  CBT can be used on ESX4.0+ with virtual hardware 7.  One or more VADP proxy system is required (Windwos 2003-2008R2) and one or more vCenter server running a supported version (vSphere5-5.1, vCenter 4.0 or 4.1 managing ESXi 3.5-4.1 or VirtualCenter 2.5 with ESX 3.5).  Networker 7.62 SP2 or newer on VADP procy and vCenter and have access to all LUNs.


VADP does not support IPv6, so disable it on Windows 2008.   Set client parallelism on VADP proxy to max number of VM backups it will run concurrently, and keep the vCenter and VADP proxy as separate machines.  Make your VADP proxy a storage node for any transport mode.

If using SAN transport mode, have 500MB RAM for each VM backup to run concurrently.  Add proxies to increase throughput.  Max concurrent backup per proxy is 50, unless using DDBoost, when it is 100.

If using Hotadd transport mode, at least 4 vCPU and 8 GB vRAM per virtual proxy.  Client parallelism should not be set higher than 12, and no more than 5 virtual proxies on one standalone ESX server.  Datastore for VADP proxy must have sufficient free BEFORE hotadd backup begins.  Can not use hotadd backup for IDE virtual disks, you need to use ndb mode for these.  Host all proxies on a dedicated ESXi server, and if VMs have more than 12 disks attached per VM, do NOT use hotadd mode.

 Oracle Integration is possible with Networker.

RMAN channels are streams of data sent to a device or app like Networker.  RMAN Multiplexing is not the same as tap multiplexing, but is rather multiplexing into a specific RMAN channel.  FILESPERSET is the max number of files for a backup set, and MAXOPENFILES determines the max number of input files a backup can have open at one time.

Tune RMAN channels by using RMAN Multiplexing.  Increase the stream count to match the target device count.  Avoid tape multiplexing.  Keep FILESPERSET <= MAXOPENFILES and increase RMAN multiplexing if there is little no striping of datafiles across disk storage and decrease if datafiles are striped across many disks.

Exclude things that don't change, but never exclude system tables (Master DB, control files) and reduce database sized by archiving databases.  RMAN backup L1 will be differential by default unless specified to be cumulative.  L1 diff backs up all data since last L0 or L1, cumulative backs up all since last L0 or full.  On Oracle 10g or newer, CBT can be enabled to increase backup performance.

FOr M$ SQL databases, stripes are like RMAN channels in Oracle.  Networker stores each stripe as a different save set. 


NDMP is used for backing up NAS devices.  NDMP separates data path from control path to minimize network resource demand.  NDMP file history is metadata sent to Networker server, and it is processed at the end of the backup job.  This file history is read into a temp file and processed on disk in the form of a B-Tree.  At the end of the backup the B-Tree on disk is processed and indexes are generated and committed.  NDMP file history processing time is influenced by storage system factors such as RAID type, number of disks, cache size, etc.

During NDMP file history processing, /nsr/tmp is heavily used and the temp file isn't removed until after B-Tree file processing completes, meaning double the space is required during processing.  Failover in a clustered Networker environment will interrupt index processing and index generation would be incomplete, so file level recovery will not be possible without restarting.

Space for the /nsr/tmp can be calculated as 2*(144+average file name length)*# of files.  for a million files with an average filename length of 128 it would be 2*(144+128)*1,000,000 = 544 MB approximately.  The location of the tmp directory can also be changed using NSR_NDMP_TMP_DIR set in the application information window on client config.




Networker Environment Tuning

To obtain maximum efficiency and system utilization, there are parameters that can be adjusted to tune the Networker environment.

Target sessions are the preferred number of sessions a storage node will accept, but it is a soft limit.  If the target sessions are met, new sessions will look for another device.  If none exist, the session count will increase to the max sessions for that device.  Max sessions is a hard limit and can bet set to a max of 512.

In Networker version 7.6 and older, the default target sessions parameter is set to 4 and the max is set to 512.  Those should be changed to 1 and 32 respectively.  In Networker version 7.6 SP1 the values are 1 and 32 per recommended value.

Networker 8.0 and newer can run dynamic nsrmmd to limit the number of backup sessions Networker server can run at one time.  It is determined by dividing max sessions by the target sessions + 4.

In environments where there is a remote Fibre Channel backup, link latency can be a significant problem.  Increasing the block size can have a drastic effect on performance because we're telling the system it can transport more data at one time.

To tune the network, TCP parameters can be changed to accommodate backup as opposed to their default, general-purpose TCP settings.  Some things to consider are to disable software flow control, increase the TCP buffer size and TCP queue depth.  Enable jumbo frames where possible and use PCIeXpress for 10Gb NICs to ensure adequate bandwidth.

Other network considerations would be to avoid network hops between clients and storage nodes.  Place a storage node on the local subnet if it is far removed from the rest of the network, or where there is a firewall to contend with.  Also consider converting larger clients to storage nodes and have them backup locally.

A backup network is another option for network tuning.  Storage nodes can be multi-homed.

Networker relies heavily on name resolution.  Servers should have low-latency access to DNS servers, so consider a local cache server or local non-authoritative name server that accepts zone transfers from the primary.  Also, avoid DNS queries all together by using a local hosts file.

Disk latency is a major consideration with Networker, just as with the other EMC products.  At 25ms and below we have optimal and stable backup.  Once disk latency hits 50 ms, backup slows down and NMC updates are delayed and fail.  At 100 ms, savegroups and sessions fail, and at 150-200 ms, Networker daemon will take a long time to launch, backup will be unstable.

If your Networker server is exceeding 100 parallel sessions, you can dedicate fast disk for the Networker databases.  RAID10 is recommended for Networker server disk storage.  If there are more than 400 parallel sessions, consider placing NetWorker databases on separate volumes.  You should disable antivirus on Networker databases, and you can use Asynchronous replication to avoid latency that comes wit synchronous replication.

Parallelism is the main tunable parameter on Networker.  You can set parallelism for the server, or for the savegrp to restrict the number of save sets that are backed up simultaneously.  You can balance the workload by running multiple groups - putting slow clients in one group separate from other fast clients.  You can also use pools to force large savesets to faster devices, however use caution to not have unnecessary pools.  You will want one for NDMP or Exchange, but not for all save groups.

Parallelism is configurable and on the servers should be set the the sum of all Target Sessions of all devices in a data zone.

Client parallelism is the number of streams from a client.  Default is 4, but you may want to tune to more or less.  The Networker server has a client parallelism setting as well to accommodate index backups, and this count should never be set to 1.  For less than 30 clients, set to at least 8.  For 31-100 set to at least 12 and for more than 100 clients, set this value to at least 16.

Group parallelism will throttle the number of clients within a group.  Keep groups at a max of 50 clients with parallelism enforced, and stagger start times to reduce load on the operating system. 

Sizing a Networker Server and Storage Nodes

Several factors affect a Networker server performance, including CPU, memory, paths and IO bandwidth and network bandwidth.  Of these, I/O is the most important because a backup server is mainly concerned with ingesting data from a client and outputting it to the backup server.  A fast CPU is good, but scalability considerations around I/O dictate that it is generally less expensive to add NICs or HBAs to increase I/O than in smaller form-factor servers with less ability to upgrade.

Sun Microsystems has determined that there is a general need for 5MHz of CPU per 1MB/second of data movement in and out of a Networker storage node.  Therefore, 1 GHz is required to move 100 MB/sec (500MHz ingest, 500 MHz to move from storage node to device).  OS typically does not play a major part in performance tuning.


Memory is determined by number of clients. Less than 50 clients only requires 4GB RAM, with 8GB required for 51-150 clients.  More than 150 clients would take 16GB, but remember to account for other apps and configuration on the OS.  Continually monitor the page file and minimize swapping wherever possible.

nsrd, nsrexecd, save each take about 16MB of memory.  savegrp requires 6.5 MB per running process.

Internal bus is the single most important component.  It ties all internal components together, and the type of bus, the data width, the clock rate and motherboard all play a part in bus speed.  When calculating bus speed, make sure you have sufficient bus width for all devices.  Look for separate buses when possible, and the more buses the better.  Faster bus does not guarantee faster performance.  Bus is normally the bottleneck, so look for PCIexpress for both server and storage nodes.

Networker catalog sizing: Catalog Size = (n + (i*d))*c*160*1.5
  • n = number of files to back up
  • d = days between full backup (how many incrementals?)
  • i = incremental data change rate
    • i = n*(% data changes as a decimal)
  • c = number of backup cycles
  • 160 = average estimate of each catalog entry
  • 1.5 = multiplier for growth
The catalog location can be changed easily.  It can go on direct attached storage, but it causes trouble with growth, or on SAN attached storage to leverage DR and growth capability.


Storage performance is measured in I/O operations per second, and if there are not sufficient IOPS, performance will degrade.  Generally, problems will occur if IOPS fall below 50% required.  To optimize for less IOPS required, spread backup time out, minimize external reporting and run index and bootstrap backup outside the backup window because they come from the Networker server itself.

Minimum IOPS for Networker server is 30, with increases coming with load.  Divide the number of concurrent sessions by 3 and add to the base to accommodate for parallelism, and add 50, 150, 400 for small, medium, large server bootstrap and index backups taking place concurrently.  Also, other factors will contribute, such as NDMP requiring a large number of IOPS due to processing, and large volume counts, or lots of servers starting backup at the same time.

Monitor I/O, disk, CPU and network using DPA, Perfmon, iostat, vmstat, netstat to monitor performance.  Look for unusual activity before, during and after backup.  Monitor over time to maintain a baseline to identify trends.

Use FTP to isolate network from tape devices.  Also, compare passive vs. active FTP to determine if packet filtering is in play.

Networker comes with 2 tools:  Bigasm and Uasm.  Bigasm tests performance of client, network and tape, removing disk delay by generating a large amount of data in RAM and transfers it over the network..  Uasm measures disk speed by actually writing data do a null device.

Networker Licensing

Evaluation licensing for Networker will run unhindered for 30 days, after which the server runs in
"enabled" mode where you load a base enabler on each Networker server and specific function enabler codes that will work for 45 days before having to authorize the product.  After authorizing the software, it is permanently available for use.

Use the NSR License resource to install codes.

The traditional model is to have an enabler for every feature and module, which is beneficial in that you only need to purchase what you use.  There are trial enablers that give access to features for 45 days.  Feature licenses activate features, such as archiving, client connection, NDMP and Data Domain.

Once a base enabler is installed, it can only be upgraded or downgraded, but not deleted.  The base enabler determines features of the Networker servers.  The 30-day eval ends when the base is installed.  The evaluation enabler extends the evaluation period 45 days, and the update enabler is required to update from 7.5 to 8.0 and is automatically generated since Networker 7.6.

Base enablers come in 4 versions:  Power, Network, Workgroup and Business.  When purchased, you receive a LAC email with enabler codes, which are combined in Powerlink with the host ID to generate an authorization code which is put into the Networker console.

nsradmin -i will install Networker licenses on the Networker server through the CLI instead of using the NWAdmin GUI, which would take place in the "Registrations" page under the NW Server.

In a clustered environment, a Composite Host ID is generated from the host ID of all servers in teh cluster.  The hostid's of each are stored in a file and Networker generates the composite host ID when restarted.

Virtual environments are licensed by the host, not the client.  Modules are licensed the same way - so all Oracle servers are covered under one plug-in as long as they are all on the same host.  VADP and VCB require one client connection license.  VCB is automatically upgraded to VADP after Networker 7.6 SP2 to accommodate the proxy.

After Networker 7.6 SP1, Networker can be licensed for all features by the capacity in TB.  Measured as largest full backup over a 60-day period.  This amount can be estimated using DPA, or EMC provides a free virtual appliance called the Asset Management and Planning appliance (AMP).

Networker Capacity licenses come in the Source Capacity Datazone Enabler, which is required for each installation and the capacity is tiered in 1TB or 10TB increments.  There are less licenses to deal with, because all features are enabled.  Requires Networker 7.6 SP1 as minimum, with no concern for client license.

Modules not included with Networker:  Module for Documentum, PowerSnap, SnapImage.

Traditional licensing is best for environments where there are few clients with lots of data.  Capacity licensing is best for environments where you lots of clients and little data, or in environments where you need lots of features.

To gather license info from Networker:  nsrlic -v <servername>  shows licenses by type, available and which clients are using them.

Licnese conformance summary is a report from the GUI that details license information.

Create a license template in Excel to keep track of licenses required as designing the system.  Add the base license last to avoid disabling features.  Don't use Networker in a prod7uction system unless auth codes are applied, and if your Networker server changes, the auth codes become invalid.








Thursday, November 7, 2013

Networker Integration

Networker integrates with VMWare much like Avamar.  It will do guest-based backup with agents installed in the guest VMs for which the VM needs to be powered on.  It will also do image-level backup using VADP using a proxy client that takes a snapshot, mounts the snap and backs up the snapshot.  This reduces overhead on the ESXi host.  Also, like with Avamar, CBT can be used to backup only changed blocks of data in the VMDK files.  Individual file restore from VMDK can take place only on NTFS volumes.

Networker integrates with cloud services, both private and public.  It will compress and encrypt the data in transit, as well as throttle bandwidth to ensure service level to applications and customers.  It will handle backup, recovery, staging and cloning.  Clones are infinitely appendable when storing in the cloud.

Networker integrates with DPA, providing expanded monitoring and predictive analysis.

Networker integrates with Avamar, making Avamar to be seen as a storage device.  It will then  use Avamar's source-based deduplication and the Avamar is referred to as the dedupe node.  One advantage over Avamar is that Networker will allow multiple simultaneous backup of the smae client, for instance you can run a file level and database backup at the same time reducing time.  All jobs are managed and configured through Networker, and all Networker Modules can be used.  One caveat is that Networker will not manage Avamar Replication.  That still needs to be configured and managed through the Avamare interface.

When a backup request is sent from Networker to Avamar, the save command spawns the avasm, which in turn spawns the nsravtar.  The process then follows the normal Avamar backup routine where it checks the hash file on the storage node, creates a hash cache and compares to new data, etc.

Data Domain can act as an AFTD or FTD, it can be utilized as VTL over Fibre Channel, and it can be used wiht DD Boost.  With DD Boost, you oobtain the benefit of central config and management of backup.  Replication is also controlled in Networker when integrated with Data Domain.  You can install DD Boost on a storage noe, or install it on the client and do client direct backup.

Networker Module for Microsoft Applications (NMM)

NMM is the application module for all Microsoft applications and databases, including Exchange, MSSQL, Sharepoint, Active Directory DPM and HyperV.  It leverages VSS-based backup and recovery, and can also use a proxy client to offload the resource use from the application servers themselves.

The backup process is started per usual with savegrp calling nsrexecd, which then calls nsrsnap, which spawns nsrsnap_vss_save, which calls the Powersnap application.  This is where the backup takes place.

Backup types for NMM are "Instant" which means to take a snapshot using VSS, and can be done witho ro without rollover, which means to take it off to tape.  It will do non-persistent rollover (no snapshot) and it will do serverless, or proxy client backup.

The recovery types are conventional, which is per normal restore from tape, and Instant, which is a recovery from a VSS snapshot.

Exchange 2007 can do Mailbox Server backup, which grabs the storage groups, mailbox database and transaction logs or individual storage group.

Exchange 2010 or 2010 DAG does the entire Mailbox Server which is all mailbox databases and t-logs or individual databases.

Exchange server will do full or incremental backup.

MSSQL 2005, 2008 and 2012 are supported since version 2.4 and there are two types of backup.  VDI, or Virtual Device Integration is more flexible.  It will do Full, incremental or differential backup, and recovery includes point-in-time, partial restores, redirected restores to another server.

The VSS backup is full every time and the database must be in simple recovery mode.

Sharepoint is a distributed server system, and NMM will backup Sharepoint 2007 and 2010 farms, content, database, web apps and search services.  Only a full backup of Sharepoint can be taken, and individual file or object restore is only possible through 3rd party applications such as Kroll Ontrack.

Hyper-V 2008 uses VSS for backing up the mangaement OS, or parent partition.  This backs up the images (VHD) files and is best used for DR only.  It is similar to the VADP image-level backup.  To get individual file restore you would need to install the agent on the guest VM itself.

Networker Module for Databases and Applications (NMDA)

NMDA is the Networker Module for Databases and Applications.  It provides integration with DB2, Informix IDS, Lotus Notes Domino, Oracle, Sybase ASE.  It is a single module that supports all those databases, and to simplify licensing replaces the individual modules that were previously available for each of those.

NMDA essentially offers integration into those applications and provides Networker functionality to their backup.

The backup process is like the normal backup, except that the savegrp is called and issues a request to nsrexecd on the client.  nsrexecd calls nsrdasv which loads the correct API for the database installed.

DB2 can backup database and tablespace, and its types are Full, Incremental and Incremental Delta.

Lotus Domino backs up the database files, DAOS and individual files.  Its types are Full, Incremental and Incremental Delta, and on Windows-installed Lotus servers there is an additional GUI for managing the plugin.

Oracle modules will back  up the database, tablespaces and data files, control files and redo files.  Its backup types are Full, Incremental 0, Differential 1 and Cumulative Incremental 1.

Networker Modules Intro

Networker modules are client programs that  allow applications to be backed up in a consistent state.  There are modules for databases, email servers, enterprise applications and snapshot management.  The main advantage of module use is that you can get online backup of application servers in a consistent state.   It also allows for 3rd party integration with those applications.

Modules are client software that is loaded on the application server directly, or a proxy client can be used to offload the resource utilization.  The proxy client could be on a storage node to allow client direct backup to that node, also.

The backup process is the same as a normal Networker backup, except that a backup API specific to the client called gets loaded and sends data to the save command, which sends the data to the storage node.

Recovery is similar, excpet the recovery command is used to call the API, which then pushes the data back into the application.

Disparate snapshot technologies can all be managed with Networker, both the backup and restore.


Networker in a Firewalled Environment

Service ports are where an application is listening for connections.  Connection ports are outbound connections to other servers' service ports.

Default service ports for Networker are TCP 7937-9936.  They listen for and service backup requests coming from a Networker server.  They are randomly chosen by the configured port range by EMC portmapper.

Connection ports are by default set to 0-0, which means use any port available.  On version 7.4 and newer, port range is set to 0-0.  Upgraded Networker servers that go to 7.4 or newer maintain the settings prior to upgrade to maintain compatibility.

Port ranges can be manually configured.  The process daemon nsrexecd will always listen on 7937 and 7938 no matter what the value is set for.  7938 is rpcbind, or portmapper and resides in /nsr/res/nsrladb on each Networker host.

Networker 8 requires the following ports:



These add to 12 for the server.  in addition, nsrlcpd requires 1 connection for each jukebox, and 2 for each FTD device connected.   AFTD requires the max count settings of all devices.

To configure the port ranges, determine the port count required.  User must have update port ranges resources permission.  Us nsradmin -p nsrexec to add permissions, or us NMC, nsrports, nsradmin to set port ranges.

NMC SErver requires 3 ports for processes and 2 for SNMP if using DD Boost.  There is 1 for the web server, 1 for RPC, and 1 for database calls.  By default, 9000 is the default httpd port, 9001 is RPC, and 2368 is the default database port.

For a firewall configuration, the following rules would need to be configured:

Outbound, TCP 7937-X from server to client
Outbound TCP 7937-X from server to storage node
Inbound TCP 7937-X from client to server

Where "X" is equal to the highest port number as determined by adding all necessary processes together