it manager salary

Exchange Server Disk Failures With Event ID 51 and 57

Just the other day I experience one of the worst system crashes that I had ever experienced with our exchange server. My Exchange Server DAG failed with event id 51 and 57 showing in the system logs.

I guess I should not call this “one of the worst crashes”.  I have experienced many in the pass.  This is normal, when working in an Information Technology department, especially working with exchange server.

Who was it that said that an Exchange Server administrator knows everything about exchange server?  Well, I for sure do not, or did I ever confessed to have known everything. Thanks to Microsoft, who always assist me when I am unable to resolve a situation as quickly as I should have. Many situation can normally be resolved, but when time is against you, researching the error thoroughly is not sometimes possible.

My Server Basic Configuration

Let me explain a little bit about my configuration first, so you understand what I am trying to explain regarding our system failure.  I have four servers in my environment.  Server A and  B are being used in my DAG configuration.  Are you familiar with Dags?  I will briefly explain.

DAG stands for Database Availability Groups. It is a build in component within Exchange Server created especially for redundancy.  Having a DAG setup in place is one of the best ways to minimize downtime and protect your system, in the event of a failure.

Most persons using a DAG configuration will say that backups are not required, as long as you have redundancy in the form of a DAG.  What if corruption of you data had replicated to all members of the DAG.  The only way to recover would have been from a backup.

Server C is used as an archive server while Server D is just there in the event I need to test an installation. Each server is setup with two controller cards so we can separate the log files from the databases.

Drive “C” on each server has the operating system with a raid one configuration.  Drive “D” is a partition on the same disk.  Drive “E” is where the databases resides in raid five (5) configuration.  This is where the failure originally took place. Server “A” was the server that experience the problem.

The Issue / Problem

Drive “E” on Server “A” where the databases resided were accessible, but we were unable to see any data.  I immediately went through a series of test including walking physically to the server room.  On arrival I noticed that only one drive on partition “E” where the database resided on was the only one showing a green light.   The remaining four drives on Drive “E” was showing no light at all.”

As I have mentioned earlier drive “E” was on its own controller card with a raid 5 configuration to protect in the event of a failed drive. Raid 5 requires that three or more physical drives are required to create the configuration.  My server had four drives in place.  As I said earlier only one drive in the configuration showed a green light, all others were dead.

On further investigations of the failed server, our system logs continually showed error messages in the form of event id 51 and 57.  These events are normally related to disk failures. These errors were later confirmed by Microsoft as being related to hardware disk failures.  Microsoft does not deal with physical system issues, so I had to open a case to HP, who created the server hardware.

HP later advised that one of the controller card had failed that the four drives were attached to.  We ran a series  of tests along with HP but could not find any physical errors with the controller card.

At first I suggested that we boot the server, but I hesitated because I wanted HP to physical check first. Eventually we booted the server and that did resolved the error. On boot of the server there was an error message acknowledging that the controller card had locked.  This could have also been as a result of the server running out of resources.  The server had about 12 GB of Ram.  I eventually doubled the amount of Ram.  The server is presently up and running.

I eventually rebuild the DAG on Server A, but did not fail-over the databases because I wanted to ensure that the server was okay.  My organization will continue running on Server B until I am satisfied that all has been resolved.

Conclusion

This day October 20, 2012, and a Friday eventing at that, will go down in history of one of my not to good days, as an Exchange Server Administrator.

The comforting part about this day is that everything happened for a reason.  If you have no problems with Exchange Server, then your will never increase your knowledge base as an Administrator. Of course as Administrators, we should never welcome problems.

Below is a final letter that I had received from Microsoft acknowledging my issues.  You may notice several links to documentation from Microsoft in relation to backups.  this is because I asked the technician to send me more information.

 

“Hello Andrew,

It was my pleasure to work with you on case# 1121———-.  As per our discussion, I will be archiving your case today as resolved . If you have any comments or questions regarding the handling of your case, please feel free to contact my manager Shivraj Chopra at 425-000-0000 Ext- 64228 or Email at: 

 Also, please remember that if you have any additional problems that are directly associated with your original issue, you may call back and have this case reopened at any time within the next 90 days.

Andrew, have a great day, and thank you for your continuing support of Microsoft products!. I am providing you with a summary of the key points of the case for your records.

PROBLEM DESCRIPTION:

====================

Passive database copies failed and change state to  FailedAndSuspended on DAG member “Server A ”

 RESOLUTION:

===========  

>Found all databases on passive node  were FailedAndSuspended except one database Exec_VP\Server A.

>Identified that healthy database was store on different drive g:\ and all effected was on E:\

>Checked event found disk related events 51&57.

>Identified issue is with hardware.

>Fixing the hardware issue resolved the issue .
 

RELATED ARTICLE:

===============       

http://blogs.technet.com/b/ehlro/archive/2010/02/13/backup-issues-and-limitations-with-exchange-2010-and-dag.aspx

 

http://technet.microsoft.com/en-us/library/dd876851.aspx

End Microsoft Letter.

 

it manager salary

A Cool Way To Extend Your Exchange Database Storage Without Any Downtime

A Cool Way To Extend Your Exchange Database Storage Without Any Downtime

Below is a live screen shoot of my additional 838 GB of additional storage space, after I managed to extend my existing Exchange database storage on one of my server in a Database Availability Group (DAG) configuration, without any down time.

Click Image To Enlarge

Advancing technology really has its advantages and has done so much for those of us in this industry.  Could you imagine trying to increase your storage without shutting down your servers?

No exchange system administrator likes dealing with system problems, but problems in data processing are normal.  Problems sometime are so complicated that it raises the stress level of any administrator to a level normally not experienced on a regular basis.

Could you imagine your boss, and many of the other employees “breading “down your neck, asking you when you are going to resolve the problem? If an Exchange administrator has no problems, then he or she will never expand their knowledge base. Problems as an exchange administrator, is evident.  In fact, as long as you are working in an Information technology department, you will experience problems from time to time.

This article will first briefly explain the problem that I had experience and how I went about resolving the issue without any interruptions to the Company.

 The Problem

We are presently moving our users from our older Exchange 2003 servers, to our new Microsoft Exchange 2010 servers.  Many of our users have already been moved to our exchange 2010 server, except for about six executives.

The existing size of the Exchange database store for the users who have not been moved to exchange 2010 servers is about 60 GB in size.  If I had to move the users that are still on our 2003 servers to our 2010 servers, then I would have definitely experience space issues.

I had to think of a way to increase my space on the 2010 Exchange servers, with minimal interruptions to the business.  Before I go any further in explaining how I increased my server space without any down time, let me tell you a little about our present configuration.

Exchange Server Configuration

My exchange configuration consists of two Exchange 2010 servers, and three Exchange 2003 servers. We have implemented High Availability, on our servers for fault tolerance, and redundancy. This redundancy is in the form of a DAG environment.  DAG is short for Database Availability Groups.

There are presently two Exchange 2010 servers setup in a DAG configuration. The mounted database is located on Server One, and the secondary or healthy database is located on Server Two. If Server one fails, then server two should automatically detect a failure and takes over.  This configuration is supported by Microsoft, so if I need assistance they normally assist me for a fee.

Both servers are HP DL 380 G6 servers, which made expanding the space much easier than using regular servers.  These servers came complete with additional expansion slots, to add additional drives, when needed.

The Resolution

As you can see we are also like many other companies still working in a mixed environment.  I am one of those administrators that came from a mixed environment, which included Exchange Server 5.5.

We never seem to be able to completely get rid of the mixed environments.  By the time you are almost off the older version, Microsoft always creates a newer version. I guess this is by design.

The first thing I did was checked the physical configuration of the server to see if I had addition slots available, to install new drives.  To my surprise, I had two slots available on both servers.

I guess I should have known this, if I had documented my server’s configurations.  Like so many administrators, I did not document our configuration. Documenting your server’s configuration, on initial install will save you many hours, when you are faced with problems.

The second thing I did was asked the purchasing agent to get me a quote on at least five 900 GB hard drives. The reason why I asked for a quote on the five drives, instead of four was to have a spare drive in the even I had a failure in the future.

Once the drives arrived, I installed them within the servers.  Two drives were installed into each server.  The reason I installed two drives in each system was to create a mirror for redundancy. If one drive failed, then the other drive would have immediately continued to work, until I was able to install my spare.

This was one of the reasons why we always purchased HP Servers.  These servers have so many build in fault tolerance, from the perspective of the hardware, instead of the software.  No need to depend on the operating system, to setup the fault tolerance.

The next thing I did was logged into each server and access the build in Raid utility, to configure the drives.

Click Image To Enlarge

If you look closely at my diagram above you will see two SAS drives that we had purchased to add to the server.  Here you will notice that the two drives are showing, but in the diagram at the beginning of this article, you would have noticed that only one drive is showing with the 838 GB.

The Array utility takes as many drives and configures them into one large drive. There is some overhead involves that is why you would have seen the 900 GB reduced to 838 GB.

Once I had configured my drives using this build in utility, they were ready for use.  My next step would have been to begin creating addition database stores using Exchange server, or Database Availability Group, for our Executives.

Once my additional stores were created then I would have begun the process of moving my Executives to the additional space.

I hope that you have enjoyed this article.  Please my other articles on Exchange server here on this site.

it manager salary

Exchange Server 2010 Prerequisites and Exchange Server 2010 System Requirements

Any wise Mail Administrator contemplating transitioning from any version of Microsoft Exchange Server, to the latest mail server version of Exchange Server 2010, would commonly be concern about the Exchange Server 2010 Prerequisites, or whatever the latest Microsoft mail server software available.

There may be later Exchange versions available, if you are reading this article several years after the published date, but the same will apply in every case. Wise administrators are developed through experience, not born with the talent to be a wise administrator.

I hope that you are not one of the few persons left in this world that are running Exchange Server 5.5.  I am just “kidding”, but really, you do need to upgrade to the latest exchange server version, so you can receive support, when you need it.

No need to feel left out, if you are still running Exchange 5.5.  I am one of those persons that refused to upgrade just because a new version of software is produced. I never like being on the cutting edge of any new software.  I love waiting until others have tried it out first. The only reason I transitioned to the latest version, was because of Industry pressure from other administrators, while attending Microsoft Tech Ed conference in June 2010.

I am sure by now, if you have been working in this field for some time, that you have heard of the term, “If it is not broken, do no fix it”.

While at the Tech Ed conference, the question was asked of participants whether they were still running Exchange 5.5, and to my surprise it was at least more than half of the attendees.  If you have never attended Microsoft annual Tech Ed conference, then you are being deprived of knowing which direction technology is headed, in reference to Microsoft products.

Another reason for upgrading to Exchange Server 2010, or later is because main line support for Exchange Server 5.5 has been discontinued, since 2005. Microsoft also has available in place extended support for Exchange Server 5.5 that began in 2006. You can imagine how expensive this can be.

Support is critical to the successful running of your exchange environment.  I am a true witness of this. Could you imagine having an email system crashed, and not having support available? Trust me; you do not want to ever be caught without some kind of support in place.  Email is a critical part of the running of any company, and any wise administrator will strive to always have support available.

AS and Exchange System Administrator you will require support at some point in your career.  No system will ever run 100% up time.  I can recall on several occasions when I needed support but was not able to received support because of no support agreements being in place. I have learned my lesson, which is one of the reasons why at the beginning of each year I would normally renew my “5 Pack Support” with Microsoft.

I have to constantly remind myself that this article is not about the day to day operation of my exchange server environment, but about Exchange Server 2010 Prerequisites.

If you are running exchange server 5.5 you must know by now that there are no direct approved upgrade paths to Exchange Server 2010.  I have used the term “approve upgrade path”, because there may be some undiscovered third party utilities available to accomplish this, but you have to be careful.

If the upgrade path that you are using is not approved by Microsoft, then they will not give you any assistance, if needed during your greatest challenge. To reduce some of the problems that you may experience during an upgrade to Exchange 2010, you should pay close attention to Microsoft representation of Exchange Server 2010 Prerequisites.

Exchange Server 2010 prerequisites require that the Active Directory Forest functional level be set to Windows Server 2003, or higher.  Whichever server is responsible for the Schema Master role within the forest, should be running Windows Server 2003 with a service pack level of at least 1.

If you are running Exchange 5.5 then you must upgrade to Exchange 2003, before you can get to Exchange 2010.  If you are running Exchange 2000, then you can easily go directly to Exchange 2010.

You should also be aware that Exchange Server 2010 will only run on Windows Server 2008 using a 64bit processor.  Exchange 2010 will not work on Server 2003 and 32bit processor.

Rather than trying to explain all of the minute details of Exchange Server 2010 Prerequisites, I want to point you in the direction of a resource that I have used when I upgraded our infrastructure to Exchange Server 2010.  Please Click Here for the document, or visit Microsoft Corporation website for more information.

Further Reading:

it manager salary

Download Exchange 2010 SP1? – What You Should Know Before Installing

Before you download exchange 2010 SP1 to upgrade your exchange server 2010 installation, there are several points to consider, before installing the software.

Are you aware that once you have installed Exchange Server 2010 with service pack 1, you will also need to install Exchange Server Roll up Update 6, for Exchange server 2010?

I am an Industry professional who has worked with Exchange Server since it’s’ inception. When I use the term since its inception, I am referring to one of the earlier versions of exchange mail server software,version 5.5.I have worked along with countless upgrades regarding exchange server, and Active Directory, just to name a few.

If you are an exchange administrator, then you should know what an exchange server is.Just in case you have accidentally accessed this article while searching for the meaning of exchange server, I will briefly explain.

Basically, Exchange Server is an email platform that serves email to users who use similar free email platforms, like Hotmail and Gmail.These free email server providers also uses some kind of email servers in their back office, to service their free email clients.

Since I have mentioned Active Directory in one of my earlier paragraphs I will briefly explain what Active Directory is used for.In fact Exchange Server will not work without Active Directory.

Active Directory is a service developed by a popular software vendor to facilitate secure access to resources on a network.Active Directory also allows the sharing of resources like printers, documents and other hardware items to users.Exchange server relies heavily on Active Directory.

Several days ago, I attempted to upgrade our current Installation to the latest release of Exchange Server 2010 software, with exchange 2010service pack 1, but experienced issues in reference to the replication service that ran in the background. The replication service “crashed” continually,almost every day.

I then proceeded to do some research in reference to the issue that I was experiencing, and discovered that many users worldwide experience this same issue.The only way that this issue could have been resolved was to install another patch.The name of the patch was Exchange Server Roll up Update 6.The lesson was that you cannot upgrade Exchange to Service Pack 1 without installing the Roll up Update.I guess this is how we all acquire experience when using a product.

If you are an exchange administrator, then you should be aware that the replication this service is required to replicate databases to other servers in the environment, when using the Database Availability Group feature of the software.

Again, if you are new to exchange server, the Database Availability Group feature is a feature that allows administrators to setup redundancy for their Servers by copying their databases to several servers in their environment.This feature allows the administrator to recover very quickly from a system failure.

Exchange Server 2010 SP1 is also required to fix an issue in reference to memory leaks, when using Windows Server 2008 Operating System.The four of our mail Servers experienced a memory leak because of the missing patches.

Because of this memory leak, my Server‘s memory allocation were improperly distributed.The task manager showed that the Memory usage continued to ran around 90% to 97% usage continually.This was not a good position to be in as an administrator.The Server potentially could have malfunctioned at any time, if the patch was not installed.

The service that was consuming about one gigabit of memory should only have consumed about half of this amount normally.If this problem was not corrected immediately then clients would have experienced very slow performances when accessing their email, with their outlook client, or other compatible mail client software.

As an Exchange Server administrator you would have also have begun to notice an abundance of RPC communications errors, recorded in the system logs, as a result of the memory leak.I will explain RPC errors in another article. RPC stands for Remote Procedure Call.

To eliminate some of the many issues that you may experience when you attempt to download Exchange Server SP1 to upgrade your existing Exchange Server 2010 environment, you should also download Roll up Update 6, the same time.

 

it manager salary

Is Exchange Sever Really The Best Email Server?

Even the best email server will experience problems every so often.  This week I experience a very minor problem regarding my exchange server.  All of our servers, at our establishment would continually show high memory usage between 90 % to 97 % on a continuous basis. When I say memory usage, I am not referring to the CPU.  My CPU was okay, running continuously between 1% and 4% usage, which was very good, and normal.

As an IT Professional you will experience issues from time to time.  If you do not experience any issues, then you will never achieve any experiences in you field.  Experiencing problems as and IT professional, are a part of our IT routines.  Please do not misunderstand.  You will have many days of “smooth sailing”.  In fact your “smooth sailing” days will surpass your problem days.

Let me tell you a little about our environment, before I explain further the problem that I was experiencing. We have four servers, or four nodes running Windows Server 2008, Enterprise addition. As I also mention in one of my earlier articles, we are running in my opinion, the best email server software, which is Exchange Server 2010.  This is Microsoft Corporation latest release, at the writing of this article.

Two of our server’s hardware are HP DL380 G6, with twelve Gigabits of memory.  The other two servers are HP also, but slightly older, with about 4 gigabits of ram installed.  The servers are running in a four node DAG configuration. The term DAG is an abbreviation of Database Availability Group, just in case you are not familiar.

At first when we noticed the problem, we taught that there may have been a fault with the server.  On further investigation we discovered that the four server were experiencing the same problems.  As you may be aware exchange server 2010 has basically five server roles.  Hub, Edge, Client Access, Mailbox, and Unified Messaging.  We have deployed four roles on each of our servers, excluding the Unified Messaging role.  At first we taught that since all services were running on each box that we had to add more memory to the systems.  We were so wrong.

I begin researching using Google and Microsoft website and came across an Article that suggested that we may have been experiencing something known as a Memory leak on each server.  This appeared to be a common problem with the initial release of exchange server 2010. The problem is more evident if Exchange is being used with a Blackberry Server.  This in fact true with our establishment.  We do have a Blackberry server installed in our environment.

I proceeded to open a case with Microsoft to assist me with the correction of this issue. That is one of the reasons I truly believe that exchange server is the best email server for all times.  Microsoft really supports what they have created.

Every year at my company, we purchase support in the form of Microsoft 5 Pack support to assist us through the year, if we experience issues.  I had two incidents remaining on our contract that was due to expire the next day.  I proceed to use the two remaining support incident with the first issue, and proceeded to open another case to attend to another issue that I was experiencing.  I was not about to let two incidents go to waste.

Microsoft really delivered and proceeded to assist me with the first incident.  The thing I love about Microsoft Support is that they will support you until the problem is resolved.

What Microsoft recommended to me was to install Exchange Server SP1 and install the roll up update 6 to the server to resolve the issues.  I proceeded to complete the tasks as they had assigned.  Presently I have install both fixes on my two older servers and the problem disappeared.  My challenge now it to install the same updates on my primary servers.

Here are the two updates that Microsoft Recommended.

Update Roll Up 6

Exchange Server SP1

I could have attempted to resolve it by myself, but what would that have profited me.  I learned a long time ago that there are no “loan rangers” in the IT field.  If you work alone you will eventually be destroyed.  So as you can see, this is why I always say that Exchange Server is the best email server, in my opinion.