Support for Modern Microsoft Exchange Environments
The Backup Exec 2012 SP2 Exchange Mailbox Archiving Option supports modern Exchange email environments such as Exchange 2007 and 2010. This includes Exchange 2010 environments using a Database Availability Group (DAG) configuration.
Note: Backup Exec 2012 SP2 does not support the archiving of Exchange 2013 environments.
For a complete list of supported Exchange versions and supported operating system platforms, please refer to the Backup Exec 2012 Software Compatibility List (SCL).
You can find a list of compatible operating systems, platforms, and applications at the following URL:
Recovery of Archived Emails
Administrator Recovery of Archived Emails
Backup Exec 2012 SP2 administrators will always have full rights to recover email data that has been archived into the vault store. This is done using the search and restore functionality of Backup Exec 2012 SP2 user interface, which allows administrators to search against all protected data managed by the Backup Exec 2012 SP2 server, including backup data and archive data.
Backup Exec 2012 SP2 administrators can recover archived emails directly back to the original Exchange server or to an alternate server, and have the option of whether or not to overwrite emails of the same name if they already exist on the destination server.
End User Recovery of Archived Emails
End users can recover archived email data using the Virtual Vault feature of the Backup Exec 2012 SP2 Exchange Mailbox Archiving Option. Virtual Vault integrates with Microsoft Outlook allowing end users to search and find archived emails and associated attachments using the search tools built into Outlook itself. This method greatly simplifies end user email recovery of archived emails, as it is done through the Outlook interface with which end users are already familiar.
The Virtual Vault appears within the Outlook interface as another mailbox ‘node’ which contains email items that have been archived from the production Exchange server’s storage to the vault store. When end users find an email in Virtual Vault representing the archived email they need to recover, they simply open the email object directly within the Virtual Vault node as they would any normal email object.
Virtual Vault within Microsoft Outlook
Email Archiving Rules
When configuring an email archiving task as a part of a backup job, Backup Exec 2012 SP2 administrators can choose from several options to ensure that archiving processes match the needs of their environment. Rules include the following additional options:
- Email age
- Email size
- Whether or not to only archive emails with attachments
- Whether or not to apply archiving rules to unread emails
Exchange Email Archiving Rules
Archive Data Copied from Backup Sets, Not Original Server
When performing archiving tasks that move old or large emails from primary storage to secondary or tertiary storage, Backup Exec 2012 SP2 does not transfer data directly from the production Exchange servers to the archive. Instead, Backup Exec 2012 SP2 scans associated backup sets managed by the Backup Exec 2012 SP2 server for emails to be archived and copies the data from the backup set to the associated mailbox archive. This moves archiving I/O to the Backup Exec 2012 SP2 server, greatly reducing the performance impact of archiving processes on production servers.
It kind of goes without saying that due to this unique archiving implementation, only Exchange servers and associated mailboxes that are backed up by Backup Exec 2012 SP2 can be archived using the Exchange Mailbox Archiving Option. If no backup jobs exist to protect a certain Exchange server, it cannot be a candidate for archiving, but you’d be amazed how many people forget that and treat the archiving bit of Backup Exec as a standalone – it is integrated after all!
Integrated Data Deduplication
Backup Exec 2012 SP2 supports two types of data deduplication. One is block level deduplication and applies to backup data captured and stored to a deduplication disk storage device. The other is Single Instance Storage (SIS) deduplication and applies to the integrated archiving technology available in the Backup Exec 2012 SP2 archiving options.
The archiving technology within Backup Exec 2012 SP2 is based on Enterprise Vault 9.0. One of the benefits of this technology is the form of deduplication known as Single Instance Storage (SIS). By default, SIS deduplication is enabled for the Backup Exec 2012 SP2 Archiving Option.
SIS deduplication technology ensures that only one copy of an email object or attachment is kept within a vault store. If an archive task processes an email or attachment that matches the fingerprint of an object already in the vault store, the archive task does not store that object again.
The SIS deduplication technology within the archiving features of Backup Exec 2012 SP2 applies across all partitions and archives within and across vault stores.
Single Instance Storage Diagram
It’s important to note that SIS deduplication technology applies only to the archiving capabilities of Backup Exec 2012 SP2 and only adds storage optimization to the vault store. Additional storage benefits can be gained by installing the Backup Exec 2012 SP2 Deduplication Option, which adds block-level deduplication (and associated storage optimization) for the deduplication disk storage device to which backup sets are stored.
Key Exchange Archiving Principles
Exchange Mailbox Archiving Option
The Backup Exec 2012 SP2 Exchange Mailbox Archiving Option will require a system mailbox to be configured on each Exchange server that is selected for archiving. The administrator will be required to enter this mailbox name in the Backup Exec 2012 SP2 interface as the system mailbox for the associated Exchange mailbox server. Backup Exec 2012 SP2 requires exclusive access to the system mailbox for archving purposes. The system mailbox is configured during the creation of a backup job that includes an archive stage and is validated before a job is submitted.
The Backup Exec 2012 SP2 Exchange Mailbox Archiving Option uses mailbox groups to manage how Exchange archiving rules are applied. Mailbox groups are configured by the Backup Exec 2012 SP2 administrator. A mailbox group is an ordered list of Exchange users, user groups, and distribution lists that belong to a specific domain. Users can belong to more than one mailbox group, but archiving settings for a particular user are applied according to the first mailbox group to which a user belongs.
Archived Email Recovery and User Access Control
For recovery security, access permissions for mailbox archives are synchronized with the original Exchange mailbox (this includes subfolder permissions). This ensures that users with access to the original mailbox are able to recover email data from the associated archive, and that users without access to the original mailbox are not able to recover email data from associated mailbox archive.
The Backup Exec 2012 SP2 administrator is always able to recover any and all archived email data using the Backup Exec 2012 SP2 interface search and recover capabilities.
Removal of Archived Emails from the Original Exchange Server
After email objects have been archived successfully, they are removed from the Exchange server resource from which they originated, thereby freeing additional primary storage space on that Exchange server.
The Backup Exec 2012 SP2 administrator can configure a vault store setting that determines when archived data is removed from the original servers. The administrator can set this option to one of the following settings:
- Delete archived item immediately after a successful archive task
- Delete archived item after the vault store has been backed up
Exchange Email Archiving Process Flow
When an archiving stage is included in a backup job, it can only archive data that has been backed up as part of that same backup job and that is stored by the Backup Exec 2012 SP2 server. When the archive task runs against data contained in these backup sets, the rules in the archive task are applied and matching data elements are copied from the backup set[s] into the vault store.
After data has been archived into the vault store, the data is removed from the original servers from which the data was backed up. The deletion of the data from the original server happens either immediately or after a vault store backup, depending upon the vault store setting.
Exchange Email Archiving Process Flow Diagram
The Backup Exec 2012 SP2 administrator can recover email data from an archive using the search and restore capabilities built into the Backup Exec 2012 SP2 user interface. This includes the ability to restore archived emails directly back to a production Exchange server.
End users can recover their own archived emails directly within the Microsoft Outlook interface using the Virtual Vault feature.
Underlying Technical Principles
Exchange Mailbox Archiving Option Basic Architecture
The archiving technology imbedded within Backup Exec 2012 SP2 is based upon Enterprise Vault and uses the same core archiving storage components found in Enterprise Vault. So it’s pretty fine technology. The storage components in EV & BE include:
- Vault store
- Vault store partitions
- Fingerprint database (Single Instance Storage (SIS) Deduplication)
The Backup Exec 2012 SP2 server manages the vault store and vault store partitions as a storage device and writes data into archives from backup data sources according to the backup jobs configured by the administrator. So the Administrator defines the archiving policies for Exchange – 90 day/200 day whatever …
Figure 2: Archiving Storage Components Diagram
The vault store is the parent container for archived data. The vault store is managed as a storage device by the Backup Exec 2012 SP2 server. The vault store is separated into partitions which contain the actual archive data. Only one partition can be open at any given time to receive new archive data. However, Archives can span more than one partition.
Vault Store Partition
A vault store partition is a path to storage (e.g. ‘E:\Archive’). A vault store partition can be in one of two states: open or closed. The partition with the open state is the partition to which new archive data is written and stored. As mentioned previously, only one partition can be open at a time, so you can only write to one partition at any one time even though archives can span across partitions.
Vault store partitions with the closed state do not receive new archive data; however, closed partitions can still be read for data recovery purposes and can have data elements deleted according to archive expiration policies configured by the Backup Exec 2012 SP2 Administrator.
An archive is a collection of archived data. For the purposes of the Backup Exec 2012 SP2 Exchange Mailbox Archiving Option, one archive corresponds to one backed up user mailbox.
Archives can have new data added to them and can have old data deleted from them according to the settings configured by the Backup Exec 2012 SP2 administrator. An archive can span more than one vault store partitition, since the partition that was open at the time the archive was created may not be the same partition that is open and receives new data for the archive at a later time.
Although the data that’s been archived is deleted at source – thereby creating storage space on the Exchange Server – the end user can still locate the archive data in the same way that they could using Enterprise Vault – dead cool frankly!
Unique Value of Backup Exec 2012 SP2 and Integrated Archiving
For small and medium size environments, Backup Exec 2012 SP2 offers a unique approach to archiving through the unification of backup and archiving processes into a single offering. By linking backup and archiving technologies into a single solution, administrators can both protect critical servers and applications for disaster recovery and also archive email and file system data to secondary or tertiary storage. Backup Exec 2012 SP2 enables administrators to realize both backup and archiving benefits while only ‘touching’ critical servers and applications once.
Integration with Enterprise Vault Technology
Backup Exec 2012 SP2 includes integrated archiving solutions for both file system data and Exchange email data. The archiving technology within Backup Exec 2012 SP2 is based on the proven, market-leading Enterprise Vault family of products. By leveraging this proven technology, Backup Exec 2012 SP2 is able to offer the following advantages to administrators:
- A single, integrated solution for both backup and archiving
- Lower total cost of ownership from using a single product to solve two key IT problems
(1) Protection of critical servers and applications for disaster recovery
(2) Controlled archiving of file system and Exchange email data to lower cost storage
- Lower impact on production servers as backup and archiving are achieved from a single ‘touch’
- Compatibility and interoperability assurance from true technology integration
Reliability from utilizing market-proven archiving technology
Archiving Process Diagram
As organizations grow and expand, upgrade paths are available that enable organizations to transition from the integrated version of Enterprise Vault in Backup Exec 2012 SP2 to the full Enterprise Vault solution.
Exchange Mailbox Archiving Option
Backup Exec 2012 SP2 licenses its integrated archiving technology through two product options: the Exchange Mailbox Archiving Option and the File System Archiving Option. This blog is designed to assist partners and customers as they design and implement Backup Exec 2012 SP2 and the Exchange Mailbox Archiving Option.
Archive … Don’t always pump for Dedupe
Not many people know about the integrated archiving option in Backup Exec – most people go for the Deduplication Option to reduce back end storage and mistakenly believe that deduplication will speed up the backups – bacause you’re storing less data … but deduplication still needs to process the data so it can actually take longer (unless you are deduplicating at the source/remote site).
If all you want to do is to speed up your backups and you are finding that the speed of the backup is primarily governed by the amount of data you are now having to shift over the network, then archiving is the puppy you’re after.
In this new series I am going to cover:
Integration of Enterprise Vault into Backup Exec 2012 SP2
- Underlying Principles and Technology
- Licensing Considerations
- Performance factors
- Best practices
If you want more detailed instucations on how to instal and manage Backup Exec 2012 SP2 and the Excahnge Mailbox Archiving Option please, please refer to the Admisnistration Guide and the Backup Exec 2012 SP2 addendum, as well as copious Tech Notes on the subject on www.backupexec.com.
Data and Storage Management Challenges
The evolution of applications and technologies designed to enable the creation, sharing, and management of data is continuing to drive data volumes higher and higher. Today, it’s easier than ever for end users to create and share data. As a result of these technologies and the dramatic data growth they facilitate, administrators are struggling to ensure their company’s critical data and application assets remain functional and protected, and are looking for ways to better manage the storage resources they employ to match the different data assets they are responsible for.
Data storage solutions come in a variety of flavors. Depending on the type, size, and priority of the associated server or application, selected storage devices may have a higher or lower capacity or have a higher or lower performance level. Some applications and servers demand high performance storage systems, while other applications and servers can be satisfied with cheaper, lower performance storage. As the performance of the storage device increases, so does its price.
Also, not all data assets are created equal. Some types of data may be accessed frequently, while other types of data may be touched only once and quickly become old or stagnant. Also, not all data assets are of the same size. PowerPoint presentations, video files, and similar types of data can be quite large. Other data types, such a Word documents and text files, are usually quite small.
Storage Efficiencies through Archiving
Archiving is an important technology through which administrators can control storage management costs. Archiving allows administrators to control what types of data reside in what types of storage. This includes controlling what data remains resident on primary storage resources — commonly expensive, high-performance storage solutions — and what data is moved to secondary or tertiary storage resources — commonly slower, long term storage devices — which might have a much higher storage capacity. Data attributes such as size and age can be leveraged by archiving solutions to help administrators control the archiving process.
By using archiving technology, administrators can better manage investments into expensive, high-performance primary storage resources by ensuring that non-critical data assets are moved to cheaper storage solutions, increasing the available capacity of primary storage in production environments and helping administrators get the most out of their current storage investments.
Archiving solutions can also improve application performance. By archiving application objects to secondary or tertiary storage resources and removing them from the original application server, application databases shrink and performance improves.
Performance Notes and Recommendations (continued)
Data Deduplication and Incremental/Differential Backups
The data deduplication technology within Backup Exec 2012 SP2 can be combined with differential and incremental backup strategies. When doing so, the reported deduplication ratio and reduction percentage may appear to be comparatively low, but this can be attributed to the smaller backup sizes captured by differential and incremental backups. Additional advantages of combining the data deduplication technology within Backup Exec 2012 SP2 and differential or incremental backups include smaller catalog sizes and increased verify job speeds.
Backup Exec Server Hosting a Deduplication Disk Storage Device
Best results are realized when the Backup Exec server hosting a deduplication disk storage device is built for performance and protected against unexpected power loss/interruption events.
Deduplication Option System Requirements
Backup Exec Server In order to host a deduplication disk storage device on a Backup Exec server, the following minimum system requirements must be met:
For Windows or Linux Agents configured to use Client Deduplication the following minimum system requirements must be met:
Note that protected clients can be either 64-bit or 32-bit versions of the platforms that Backup Exec supports.
The Backup Exec server has other detailed requirements listed in the Backup Exec 2012 Administrator’s Guide. Be sure to refer to the Administrator’s Guide and the Backup Exec 2012 SP2 Administrator’s Guide Addendum before configuring deduplication disk storage devices.
Deduplication Best Practices Articles
Additional notes and best practices around Backup Exec 2012 SP2 deduplication technology can be found using the following links:
- Deduplication best practices: http://www.symantec.com/docs/HOWTO74446
- Deduplication in a CAS environment: http://www.symantec.com/docs/HOWTO74447
Performance Notes and Recommendations
Configuring Backup Exec’s Deduplication Disk Storage Device
Backup Exec 2012 SP2’s Deduplication Option allows customers to create a single deduplication disk storage device per Backup Exec server. A deduplication disk storage device is where all deduplication blocks are stored, regardless of whether client-side or Backup Exec serve-sider deduplication was used for a specific backup. A Backup Exec server hosting a deduplication disk storage device can hold up to 32 TB of deduplicated data. Note that the deduplication disk storage device is not utilized for backups and restores when appliance deduplication is used. Appliance deduplication stores all backup data on the specific appliance.
Processor Utilization with Client and Server Deduplication
Depending on the type of deduplication used, processor utilization will vary. In general, the deduplication process is not gated or throttled in any way, and is geared towards accomplishing deduplicated backups and restores of deduplicated data as quickly as possible.
Client-side deduplication performs the bulk of deduplication calculations on the client (or source) system. The client-side deduplication process will consume as much of one (1) core of one processor as it can on that client system during deduplication calculations. While the actual amount of processor utilization will depend on the amount of data to be deduplicated and the speed of the processor, expect to see at least 75% processor utilization for that processor core for the duration of deduplication processing. During catalog operations, CPU utilization on the client is very low.
Backup Exec server-side deduplication performs the bulk of the deduplication calculations on the Backup Exec server. Similar to client-side deduplication, the Backup Exec server-side deduplication process will consume as much of one (1) core of one processor as it can on that Backup Exec server system. While the actual amount of processor utilization will depend on the amount of data to be deduplicated and the speed of the processor, expect to see at least 75% utilization for that processor core for the duration of any Backup Exec server-side deduplication backup job. For both client-side and Backup Exec server-side deduplication, initial backup jobs will be the slowest. As time goes on, backup speeds increase as more database fingerprints are created within the database.
For agents that cannot use client-side deduplication (Agent for Mac, etc.) there is no change to system requirements as outlined in the Backup Exec 2012 Administrator’s Guide. This is also true for Windows and Linux Agents that do not choose to use client-side deduplication.
Memory Utilization with Client-side and Backup Exec Server-side Deduplication
With both client-side and Backup Exec server deduplication, the majority of memory consumption takes place on the Backup Exec server. This is primarily a performance optimization geared towards fast and accurate calculation of deduplication fingerprints. On the Backup Exec server, both client-side and Backup Exec server-side deduplication require 8 GB (gigabytes) of free physical memory for the first 5 TB (terabytes) of deduplicated data stored by the Backup Exec server. For storage beyond 5 TB of deduplicated data, a formula of 1.5 x N, where N is the total number of deduplicated storage in terabytes, is used to calculate free memory requirements. For example, if 8 TB of deduplicated data is stored, the Backup Exec server would require at 1.5 x 8 GB, or 12 GB, of free physical memory.
Memory requirements for clients using client-side deduplication are not very stringent. Symantec requires 1.5 GB of free physical memory on each individual client that uses client-side deduplication.
Data Deduplication and Granular Recovery Technology
Backup data stored on a deduplication disk storage device hosted by a Backup Exec 2012 SP2 server or a Backup Exec 3600 Appliance is fully compatible with Backup Exec’s powerful Granular Recovery Technology. This technology enables administrators to recover granular objects, such as files, folders, and granular Exchange, SQL, SharePoint, and Active Directory components quickly and easily.
For more information about Backup Exec 2012 SP2’s powerful Granular Recovery Technology, visit the Backup Exec product page at www.backupexec.com.
In addition to its industry-leading deduplication technology, Backup Exec 2012 SP2 allows administrators to make copies of deduplicated backup sets and transfer them to a different Backup Exec 2012 server or Backup Exec 3600 Appliance. The transfer process is optimized, meaning only unique blocks – those not already contained in the deduplication disk storage device of the destination server – are copied. This ability is known as optimized duplication.
In order to enable optimized duplication, the Backup Exec Deduplication Option must be licensed on each Backup Exec server involved in the transfer process. Copying backup sets from one Backup Exec server to one or more additional Backup Exec servers using optimized duplication makes the same backup data available for recovery at multiple locations, thereby offering a convenient disaster recovery solution. Optimized duplication works just like a duplicate stage from “disk to tape” or “disk to disk to tape”, but the destination is the deduplication disk storage device on another Backup Exec server and is optimized by deduplication technology. Some additional advantages of optimized duplication include the following:
- Avoids rehydration of data while transferring it over the network or WAN
- Copies only unique data blocks not already contained by the destination Backup Exec server
- Transfers backup data significantly faster than traditional or non-optimized copy events
The benefits of optimized duplication can be very dramatic in subsequent backup operations in situations where data change rates are low.
The Backup Exec 2012 SP2 Deduplication Option is licensed per Backup Exec server. For each Backup Exec server in an environment that requires deduplication capability, an individual license of the Backup Exec Deduplication Option is required.
For Optimized Duplication scenarios where backup data is transferred between two Backup Exec servers, each Backup Exec server must be licensed with the Deduplication Option.
Example Licensing Environment
Here is an example Backup Exec environment that needs to be enabled for deduplication:
Deduplication Option Example Licensing Diagram
This example environment includes two Backup Exec servers that need to be enabled for deduplication. As the Deduplication Option is licensed on a per Backup Exec server basis, in order to enable deduplication on both Backup Exec servers two Deduplication Option licenses would be required, as per the chart below
In this same example environment, two licenses of the core Backup Exec 2012 SP2 product, one for each backup server, would also be required. Other licenses may also be needed for additional capabilities, such as the Agent for VMware and Hyper-V to enable image-level backups and granular recovery of VMware virtual machines, and the Agent for Applications and Databases to enable granular application recovery. To unlock all Backup Exec 2012 SP2 capabilities in the above example environment, the following licenses would be required:
Backup Exec 2012 SP2 2, Deduplication Option -2, Agent for Applications and Databases 3, Agent for VMware and Hyper-V 1 Agent for Windows 1.
Additional licenses may be required to support advanced functionality, such as the replication of deduplicated backup data between Backup Exec servers (known as optimized duplication). In order to enable optimized duplication, an Enterprise Server Option license is required.
Data Deduplication and Virtual Machine Backups
The Backup Exec 2012 SP2 Agent for VMware and Hyper-V enables optimised, image-level backups of both VMware and Hyper-V virtual machines. This is accomplished by capturing backups through communication with the VMware or Hyper-V virtual host. However, deduplication recommendations differ between VMware and Hyper-V environments being protected by Backup Exec 2012 SP2.
Deduplication of Image-level VMware Backups
While it is possible to utilise client-side deduplication when protecting VMware virtual machines, this configuration requires that backups be processed by locally installed agents within the virtual machines themselves (either the Agent for Windows or the Agent for Linux). This configuration bypasses the optimised, image-level backup capabilities of the Agent for VMware and Hyper-V in VMware environments that take advantage of Backup Exec’s advanced integration with the VMware vStorage API. For this reason, using client-side deduplication in VMware environments is generally not recommended. Backup Exec server-side deduplication is optimal.
Backup Exec Server-side Deduplication of VMware Image-level Backups
Deduplication of Image-level Hyper-V Backups
Client-side deduplication can be used when capturing image-level backups of Hyper-V virtual machines using the Agent for VMware and the Agent for Microsoft Hyper-V. In this configuration, optimised, image-level backups of virtual machines are captured and deduplicated through the Backup Exec Agent for Windows installed locally to the Hyper-V host. It is not necessary to install an individual agent into each Hyper-V virtual machine in order to realise client-side deduplication in Hyper-V environments.
Client-side Deduplication of Hyper-V Image-level Backups
VMDK and VHD Stream Handlers
Backup Exec 2012 SP2 includes stream handler technology designed specifically for image-level backups of VMware and Hyper-V virtual machines captured through the Agent for VMware and Agent for Microsoft Hyper-V. The stream handler technology within Backup Exec operates invisibly, meaning no additional management or configuration adjustments are required on the part of the administrator.
The stream handler technology within Backup Exec applies to both client-side and Backup Exec server-side deduplication. The stream handlers enable variable-length segmenting of VMware (VMDK) and Hyper-V (VHD) disk files during deduplication calculations. This aligns deduplication blocks to file extent boundaries within the virtual disk, and data changes over time within virtual disk files result in fewer unique blocks. This translates into better storage savings across both VMware and Hyper-V backups when using the Backup Exec 2012 SP2 Agent for VMware and Agent for Microsoft Hyper-V in conjunction with the Backup Exec 2012 SP2 Deduplication Option.
Combining the Agent for VMware and the Agent for Microsoft Hyper-V with the Deduplication Option can offer significant storage savings for Backup Exec administrators, allowing them to reduce storage costs by getting the most out of the backup storage resources at their disposal.
Additional information on the Backup Exec 2012 SP2 Agent for VMware and the Agent for Microsoft Hyper-V can be found in the Backup Exec 2012 SP2 Agent for VMware and Hyper-V Technical Feature Brief, and in the Backup Exec 2012 Administrator’s Guide and the Backup Exec 2012 SP2 Administrator’s Guide Addendum.
Data Deduplication and Storing Backups to Tape
An environment with a disk-to-disk-to-tape architecture is fairly common among customers who are interested in deduplication. It’s important to note that all of the Backup Exec 2012 SP2 methods of deduplication mentioned here are disk-based; deduplicated data is never stored directly to tape in its deduplicated form. However, the process of migrating deduplicated data to tape is very simple. Customers simply add an additional stage to their backup workflow that sends the data to tape storage.
Data Deduplication and Storing Backups to Tape
For data that was backed up using client-side or Backup Exec server-side deduplication, the Backup Exec server is responsible for “rehydrating” the deduplicated data – meaning the process of recreating whole files from deduplicated blocks – before transferring the data to tape. There will be some impact to processor and memory usage during the tape stage of the backup workflow due to the rehydration process. While resource consumption varies based on data set, at most the tape stage of the backup workflow will use 100% of one processor core while rehydrating deduplicated data and copying it to tape.
For data that was backed up to a deduplication appliance, the deduplication appliance itself is responsible for “rehydrating” the deduplicated data prior to it being sent to tape.