Monday , June 26 2017

How to restore Exchange Databases from a Storage failure

We should look for options when “Exchange Databases are not mounting”

  • Good backup Available.
  • No Backup – Exchange mailbox Databases are down.
  • Mounting a blank database – Keeping the messaging alive until you repair the exchange databases.
  • Soft recovery and Repairing the Exchange Databases
  • Recovery Databases

Have tried to include as much scenarios as possible.

Good backup Available

Best option is restoring from Backup software’s like Symantec which is the best option to have minimal downtime. but make sure you have to retain the live data whichever is on the existing drives – As you will lose data from the Backup taken time to the Storage Failure time.

if you have a good backup First Restore from the Backup get the production running. Then you can create a recovery database and repair the broken databases and merge with your production.

you may have to take a copy or rename the database file before restoring using the backup software as they can overwrite the databases files residing on the existing drives.

No Backup – Exchange mailbox Databases are down

Some Organizations start depending on DAG . Backup less solutions . Still there are chances where your Database didn’t failover due to a network issue or various other reasons. Try forcing the failover. Check you can access the storage and bring that back. The stability of the databases have increased where the server or storage undergoes a intermittent failure new exchange version cure themselves in few scenarios when it comes to database copies.

If you don’t have a backup , repairing the existing databases takes time approximately 5-10 Gb per hour.Totally depends on the IOPS/processor. if you don’t have a backup always take a copy of the broken databases. so that even if your repair is interrupted you don’t lose hope,you can copy it again for recovery purposes . Before that,There are various ways to get your databases healthy, 5-10 gb per hour is the worst case scenario.

You got to check the health of the database , Where Exchange cannot connect back to a database again . if its not gracefully dismounted or disconnected from storage or server or anti virus removes sometimes hold the log files mistakenly. lets see how to check the health of the databases.

Open powershell

Locate to the .\eseutil.exe,Default location –

Cd "C:\Program Files\Microsoft\Exchange Server\V15\Bin"

To Check the status of the Exchange database :

locate to the bin folder to check the health of the exchange database

.\eseutil.exe /mh "D:\log files\Mailbox Database\Mailbox Database.edb"

there are two results, It may say clean or dirty , will go through both.

showing CLEAN SHUTDOWN –  Database is healthy and its in a good shape. It couldn’t mount as its not able to understand the existing sequence of the log files.

Removing all the logs files from the logs files location and Mount the database. It should generate a new series of log files and mounting the database gracefully.

image

To get log file Location –

Get-mailboxdatabase | fl Name,*path

To force mount the databases –

Get-mailboxdatabase "Database Name" | Mount-database –Force

showing Dirty SHUTDOWN –  Database is not in a good state (worst case) , if the database sizes are massive, you cannot keep the environment down until the databases are repaired. here is the trick of mounting a blank database to keep the environment active going with a blank mailbox. and repair the broken databases and swap it again then merge them. if you don’t want to mount a blank database.skip it.

Mounting a blank database – Keeping the messaging alive until you repair the exchange databases.

  • Stop Microsoft Exchange Search Service
  • Stop Microsoft Exchange Search Host Controller Service

image

Now you can rename the database folder. Create a identical folder.

image

Mounting the store will force the creation of an empty database.

image

As soon as you mount a blank database in the messaging environment. Outlook will prompt for a restart. Once the outlook is restarted.

Users get an option of getting into Temporary mailbox to send and receive emails or Use Old data to look their cached PST ,if they are in Outlook cache mode.

image

if health check shows dirty –

.\eseutil.exe /mh "D:\log files\Mailbox Database\Mailbox Database.edb"

Lets see how we can handle the broken database. you can see a row called “logs required”

Check the required logs are available or not . in my case its 6079 – 6104

image

Check if you have the logs available

image

if you have the logs available – Make sure you got the .chk file in the same location. If you don’t have the required log files skip this step.

You can try running the soft recovery –  (/r)

Have the database and log files in the default location

If you cannot have in the default location use /D for database location , /s for checkpoint file location , /l for log file location.

if you don’t mention it. it will take the default location

“/a” is for – even if some log files are missing it will try to get the database to a good shape (Data loss will be there)

E01 – Go to the log file location check how it starts E00 or E01 or E02

.\eseutil.exe /r e01 /l "D:\log files\Mailbox Database" /d "D:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database\Mailbox Database.edb" /a

Check the status after the soft Recovery – If its showing Clean Shutdown – You can mount the database, if it doesnt , you can always move the logs and try mounting it as the databases are in clean shutdown.

If its not in Clean shutdown . Even after soft recovery Process

Repairing the Exchange Database : (5 to 8 GB /hour) (Exchange 2010 and later versions are much faster)

.\eseutil.exe /p "D:\Databases\Mailbox Database\Mailbox Database.edb"

It will repair the database with 98% of success – Where data loss will be there in the corrupted portion of it. Mostly its minimal.

Once the repair process is completed.We can see the database to Clean shutdown .

I would recommend to get the mailboxes moved to a different database as soon as possible , to be on a safer side. also the Microsoft supportability point of view.

Recovery Databases  –

when its comes to recovery databases,you have to understand about database swapping as well.if you repaired a 500 Gb databases and your temporary database is 5 gb. there is no point in merging 500 gb recovery database with 5 gb temporary database. Also the outlook will always wants the old database back in place to overcome the initial prompt when you have a temporary database mounted. As you dismount the blank database and mount the repaired database as primary and smaller database on the recovery . So that merging can be done quicker and simpler.

Creating a recovery database with existing database –

New-MailboxDatabase -Recovery -Name RDB -Server mail -EdbFilePath "C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Recover\Mailbox Database.edb" -LogFolderPath "C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Recover"

image

Merge them for one mailbox –

New-MaiboxRestoreRequest -SourceDatabase "RDB" -SourceStoreMailbox "vaishika sathesh" -TargetMailbox vaishika@careexchanged.in

If you have different users with same display name – below command should help you.

New-MailboxRestoreRequest -SourceDatabase "RDB" -SourceStoreMailbox 3965757c-f9f4-4c8d-a8c2-56a410257b52 -TargetMailbox vaishika@careexchanged.in -SkipMerging StorageProviderForSource

Merge them in bulk –


Foreach ($mailbox in Get-MailboxStatistics -Database RDB)

{

New-MailboxRestoreRequest -SourceDatabase RDB -SourceStoreMailbox $mailbox.DisplayName -TargetMailbox $mailbox.DisplayName

}

Known Errors :

he call to 'net.tcp://mail.careexchange.in/Microsoft.Exchange.MailboxReplicationService mail.careexchange.in(15.0.620.24 caps:3F)'
failed. Error details: must be logging in with GUIDs, not legDN
Parameter name: owner.
+ CategoryInfo : NotSpecified: (:) [New-MailboxRestoreRequest], CommunicationErrorTransientException
+ FullyQualifiedErrorId : 436EF2E5,Microsoft.Exchange.Management.RecipientTasks.NewMailboxRestoreRequest
+ PSComputerName : mail.careexchange.in

Resolution –

Use


Foreach($mailbox in Get-MailboxStatistics -Database RDB)

{

New-MailboxRestoreRequest -SourceDatabase RDB -SourceStoreMailbox $mailbox.DisplayName -TargetMailbox $mailbox.DisplayName -SkipMerging StorageProviderForSource

}

Now Consider Database Repair Failed –

Mount a blank database.

Go to the cached outlook – Export to PST via Outlook.

Create a new Outlook Profile – Import PST

Other options – you can consider 3rd party solutions for EDB to PST conversion.
After repairing the databases if users have issues in accessing folder you can run a repair on the mailbox

New-MailboxRepairRequest -Mailbox vaishika@careexchanged.in -CorruptionType ProvisionedFolder,SearchFolder,AggregateCounts,Folderview

Known issues –

Issue :

Unable to submit online integrity check request for database Mailbox Database. Failure:An unexpected error occurred

while trying to submit the request. Detailed error information follows

Microsoft.Mapi.MapiExceptionInvalidParameter: MapiExceptionInvalidParameter: Queue online isinteg request failed.

(hr=0x80070057, ec=-2147024809)

Resolution –

Database is in bad share after repair – Create a new database and move them

Issue :

Log Name:      Application
Source:        MSExchangeIS
Event ID:      2006
Level:         Error
Description:
Microsoft Exchange Information Store worker process (12584) has encountered an unexpected database error (Illegal duplicate key) for database ‘Mailbox Database’ with a call stack of
at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetTableOperator.Insert(IList`1 columns, IList`1 values, Column identityColumnToFetch, Boolean unversioned, Boolean ignoreDuplicateKey, Object& identityValue)
at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetInsertOperator.ExecuteScalar()
at Microsoft.Exchange.Server.Storage.PhysicalAccess.DataRow.Insert(IConnectionProvider connectionProvider)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.ObjectPropertyBag.Flush(Context context)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.Item.Flush(Context context)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.Message.Flush(Context context)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.Message.SaveChanges(Context context)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.TopMessage.SaveChanges(Context context, SaveMessageChangesFlags flags)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.SaveChangesInternal(MapiContext context, MapiSaveMessageChangesFlags saveFlags, ExchangeId& newMid)

Log Name:      Application
Source:        MSExchangeIS
Event ID:      1046
Level:         Error

Description:
Unexpected error encountered in critical block. Location:(Microsoft.Exchange.Diagnostics.LID), scope: (MailboxShared), callstack: (   at Microsoft.Exchange.Server.Storage.StoreCommonServices.Context.OnCriticalBlockFailed(LID lid, CriticalBlockScope criticalBlockScope)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Context.CriticalBlockFrame.Dispose()
at Microsoft.Exchange.Server.Storage.LogicalDataModel.TopMessage.SaveChanges(Context context, SaveMessageChangesFlags flags)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.SaveChangesInternal(MapiContext context, MapiSaveMessageChangesFlags saveFlags, ExchangeId& newMid)
Log Name:      Application
Source:        MSExchangeIS
Event ID:      1002
Level:         Error

Description:
Unhandled exception (Microsoft.Exchange.Server.Storage.Common.DuplicateKeyException: JetTableOperator.Insert —> Microsoft.Isam.Esent.Interop.EsentKeyDuplicateException: Illegal duplicate key
at Microsoft.Isam.Esent.Interop.Server2003.Server2003Api.JetUpdate2(JET_SESID sesid, JET_TABLEID tableid, Byte[] bookmark, Int32 bookmarkSize, Int32& actualBookmarkSize, UpdateGrbit grbit)
at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetTableOperator.Insert(IList`1 columns, IList`1 values, Column identityColumnToFetch, Boolean unversioned, Boolean ignoreDuplicateKey, Object& identityValue)

Resolution –

Database is in bad shape after repair – Create a new database and move them

About Satheshwaran Manoharan

Satheshwaran Manoharan is an Microsoft Exchange Server MVP , Publisher of CareExchange.in Supporting/Deploying/Designing Microsoft Exchange for some years. Extensive experience on Microsoft Technologies.

Check Also

SCCM Updates and Services Keep Downloading

Log Files Location – C:\Program Files\Microsoft Configuration Manager\Logs\dmpdownloader.log — Logs –  Failed to call Initialize. ...

4 comments

  1. need to understand, So after “Use Temporary mailbox” Use Original Mailbox” is it going to create a mailbox for user in newly created empty databse if user selects use original mailbox?

    what happens if user select use temporary mailbox?

    also i believe in this case you need to use third party tools to extract all the mailboxes to a pst from edb file? like we used to use exmerge in earlier versions or quest tools etc … correct?

    pls reply.

    thx

    Mk

    • Temp mailbox – gets you to empty mailbox (new one )

      Old mailbox – gets you to the disconnected mailbox (cached data in outlook )

      Restore the database
      To get rid of this

      Or

      Re create outlook profile
      After taking PST out from the old cached profile
      Or by diff means as yu mentioned

  2. And the more peaceful I felt, the clearer I seemed to be able turn out
    to be on a few things i wanted for my young
    man iq test i always admijt I sometimes tthe absence Scs test guide Put everything away that reminds you of she boyfriend nclex-pn test Understand
    what will speak that opportunties report of engineer network
    marketing The most impoortant book is old standby test guide
    – Harrison,

  3. And the file manager arranges and organizes everything in the assortment!
    Isn’t itt a fantastic feature? Questions And
    Ansdwers To Interview Questions (Zoila) bbut let’s see what the good news iss paramedic education My parents had aready separated but hadn’t signed the
    divorce papers at this poin study guide your write-offs can eeven include things like your
    car certification exam There are they who will understand and tthe wonderful that will
    not questions unanswered The bad thing about food items is you actually will workout on an earloier version off Windows nine
    pmp certification exam

Leave a Reply

Your email address will not be published.