C3S

Two days ago I decided to move our BE2010 database to a new folder on our SQL server. This didn’t go well, probably because I didn’t follow best practice but it was something I’ll learn from. In conjunction with a maintenance renewal from Symantec that appeared to be missing some of our agents this has kicked me into doing two things, firstly making sure our license and maintenance are all done as one purchase order (currently they are spread across the year) and secondly moving our production backup infrastructure to BE2012.

That was two days ago….

Rather than upgrade I decided that a clean server build would suite our purposes and so we now have newly built, fully patched server with BE2012 sitting on top of it. I struggled with BE2012 at first but I am really beginning to like it. The interface is a huge improvement. The workflow for settings up jobs is much more intuitive.

There is one problem thought and this really only applies to anyone who is using Enterprise Vault. In previous versions selecting EV was a relatively straight forward process. In the latest version there is no agent and, rather than having a clearly labelled selection list, I now have something called “Directory on xxx“. Not nearly as obvious. Only when you enable the column ‘Data Source Types’ on the Backup and Restore screen do you see that this is indeed the Enterprise Vault data.

Personally I think the Data Source Types column should be enabled by default.

Note. This was meant to go out a few weeks ago before I went on holiday. A lot has changed since then but the info in this post is still valid……

Yesterday I finally got round to building a test VM and installing BackupExec 2012 on it. Our existing backup system is now back to running 2010R3 and I have valid backups for last weekend so all is looking better.

New server is 2008 R2  1xE5440 4GB VM with a 40GB System drive and a 3TB iscsi target for storage. Installing 2012 was a breeze on a clean system (I’ve gone for a local SQL Express instance for the trial but will use a standalone SQL server for the production environment). Post install the LiveUpdate run successfully install the new hotfix and there I with the system ready to backup some data!

The interface I have pretty much got used to now and it’s beginning to make more and more sense. Disk storage was automatically configured during installation/first run – my one gripe about this is the default setting of one concurrent connection.

Setting up backup jobs is relatively straight forward, just point the wizard at the machines you want the agent installed on and off it goes. For the most part I found a reboot was unnecessary on machines where the agent was being upgraded to from 2010Rx as simply restarting the agent service on the target completed the install.

Creating the backup jobs and selection lists is all one task now but this is where I first started to encounter a few problems. The first server I choose to backup was one of our domain controllers. The agent install appeared to go fine but when I went to configure a backup job I repeatedly get “Remote Agent Not Trusted”. I’ve put this to one side as I don’t wan to get sidetracked by one issue at this point.

Next up was a fileserver. We use DFSr and, in the past, I’ve enabled backing up folder shares as backing up from the DFSr shadow copy was too slow for the backups to complete within the backup windows. However, in this case I thought I’d try backing up DFSr the “proper” way.

266GB backed up in 7 hours and 16 mins at roughly 840MB/min. This is significantly faster than BE2010

Or back to 2010 R3.

Last night I reverted our backup infrastructure to 2010R3. And by revert I mean uninstall 2012 and start from scratch with 2010R3. I’ll point out right at the start that I should have built a test system first so that I could understand all the fundamental changes to the system and also confirm that the agents worked as expected.

Compared with previous version upgrades this was a completely different experience as 2012 switches the focus away from backup jobs and rather focuses on the assets backed up. Once I got my head round this I quickly grasped the new aspects of the system and actually liked the new product.

But I couldn’t get the Enterprise Vault agent to work, at least not in a way that I was sure was working. In 2010R3 you could select Enterprise Vault from the selection list and everything pertaining to Enterprise Vault is backed up. A great way for the agent to work. Everything is taken care of in one stroke and a sysadmin can move on to worrying about other aspects of the infrastructure.

This “One click protection” appears to be missing from 2012. This may be my mistake but after several hours research and consulting the manual I was still unsure it the test backup was backing up all the Enterprise Vault data. This wasn’t a great state of affairs so in the end I uninstalled the server and agents and redeployed 2010R3 and it’s agents.

Next week I’ll build a test server and see if we can get a little further.

I’d also like to mention that Symantec’s BackupExec twitter presence has been very positive even in the face of less than positive feedback from myself.

So after reading the dire warnings displayed by the installer I have plunged ahead with the deployment. All seemed successful and I was soon greeted by the Finish button. After rebooting the LiveUpdate package kicked in automatically, a nice touch I thought, and lo and behold there is a patch to download and install. Except it doesn’t. Sadly, while Symantec may have worked long and hard on their new product it hasn’t stretched as far as delivering constructive error messages to the system administrator.

“LU1812: A program that was part of this update failed when it ran.  This update was not applied.”

All too familiar. Sadly the event log doesn’t really contain any more useful information.

Backup Exec Alert: Backup Exec LiveUpdate Error
(Server: “C-BE01″) Update HOTFIX 182395 (X64) failed to install.  Review the Backup Exec installation log for more information.

We’ll leave this problem for a moment and actually run the administration console for the first time…….

So some progress has been made. Currently I am testing a full GRT backup after enabling write caching on the SAS array we use as the primary disk target. This does appear to have bumped the speed up to approx 340MB/min. For me that just barely qualifies as acceptable performance.

However, the real shocker has been the performance of our tape drive.

As a test the Symantec Tech rep asked me to reconfigure one of the jobs for D2T instead of the usual D2D2T configuration I normally use. The flat database backup ran at about 4700MB/min, about 1700MB/min faster than our D2D of the same database. The GRT job was even crazier – this ran at 4300MB/min! Even if we don’t get better speeds during D2D backups I can dedicate one of our tape drives just to exchange backups and they’ll complete in a timely fashion.

Testing will continue with the help of Symantec’s Enterprise Support who have been very good but i have to say that this has changed my perception of how our backup infrastructure performs.

More later (maybe)

Do a search on the above and you’ll get quite a few hits. Add the word ‘slow’ in there and you’ll get pretty much the same result.

We have an exchange environment currently consisting of 4 mailbox servers, 1 hub server and a UM server. Over the past couple of months (in fact since BackupExec 2010 R3 SP1 arrived) our exchange backups have started running extremely slowly. I’ll qualify that by saying that the backup jobs with granular recovery technology enabled have started running slowly. If we just backup the entire database and logs we get speeds of around 2000-3000 MB/min. That’s pretty cool. Less than 2 hours for each server. But if we enable GRT the speed drops down to 70-100MB/min. 15 hours per server.

15 HOURS!

So I am currently on the phone to symantec support. We’ve been through all the basic options and it turns out I haven’t overlook any obvious settings.

After some research this evening I have finally tracked down the cause of the problem.

Environment : Exchange 2007, Mitel 3300, Outlook 2007

Symptoms : User has received a voicemail but, when they dial the Outlook Voice Access System, they are told there are no voicemail messages.

Cause : Corrupt Voice Mail search folder in the users mailbox.

Resolution : Delete the old Voicemail search folder and recreate. Follow the steps below:
1. Log onto user’s machine via LogMeIn.
2. Open Outlook and modify account settings to operate in Non Cached Mode.
3. Close and then reopen Outlook for changes to take effect.
4. Delete the Voice Mail folder from the Search Folders.
5. Create a new search rule called Voice Mail and configure to search to the words Voice Mail in the subject line.
6. Click on the search folder to list all voice mail messages in the user’s mailbox.
7. Call the Outlook Voice Access system and log on as the user to check voicemails.
8. Set Outlook back to cached mode.

This is a rehash of an original post on Elan Shudnow’s post here

This is pretty much the email I sent out to our support team

Combination of playing around with a few flash components and publishing some more pics from my recent trip to Cambodia.
Kep and Kaoh Poah, 2011/10/11