Failure to Restore SQL Server Transaction Log (Data Replication Project)

In Data Replication Project, there was a need to simulate recovery in event of server crash. To recover as much data as possible, we needed to restore transaction log files.

Challenges:

• Number of transaction log files were many and not consistent (some DB’s have only 10 and some may have 450+ transaction log files)
• Sizes of Transaction log files vary
• It should be an automated job that in event of server crash, jobs needs to be run to restore full backup and followed by transaction log backup.
• Naming convention of backup files was not consistent. If an administrator had taken backup manually there are chances that it would not have same naming convention as is done through job.
• Time stamps were not consistent (not sure how they ended up with this) but if
  • TLog_1 had modified date timestamp of 2/10/2012 12:00 PM
  • TLog_2 had modified date timestamp of 2/9/2012 09:00 AM.
  • TLog_3 had modified date timestamp of 2/10/2012 01:00 PM
• If you look above files closely restore sequence is TLog_2 followed by TLog_1 and then TLog_3

To automate process of restoring a stored procedure has been created that iterates through all TLog backup files (.trn) in a directory and restores them by time order and not name order. Below is code snippet for restoration.

CreateProcedure USP_Restore_Logs
    @BackupFolderPath nvarchar(1000) = null,
    @StandbyFilePath nvarchar(1000) =null,
    @BackupfileNamePattern nvarchar(100) = null,
    @DBName nvarchar(1000) = nullasBegin
--Checkinputparametersif (
            len(@backupFolderPath) = 0 or
            len(@backupfileNamePattern) = 0 or
            len(@DBName) = 0 
        )beginprint'Provide Backup Folder Path, Backup File Name Pattern and DB Name'returnend
--Declartion of VariablesDeclare @Query nvarchar (max) =nullDeclare @BackupPath nvarchar(1000)Declare @BackupFile NVARCHAR(500) Declare @cmd nvarchar(1000)Declare @NoOfRecords intDeclare @FileList TABLE (id intidentity(1,1),backupFile NVARCHAR(255)) 
--Variable InitalizationSET @BackupPath = @BackupFolderPath + @BackupfileNamePattern + '.trn'print @BackupPathSET @cmd = 'DIR /b /OD ' + @backupPath 
--Getfile list ordered bydate
    INSERT INTO @FileList(backupFile) EXEC master.sys.xp_cmdshell @cmd 
--Ifnot files we get filenot foundand a null record alwaysdeletefrom @FileList where
    backupFile isnullor
    backupFile like'%File%not%found%'
--Get number of records    Select @NoOfRecords = count(*) from @FileListif(@NoOfRecords = 0 or len(@NoOfRecords) = 0)beginprint'No File found'returnend
--Cursortorestore filesDECLARE backupFiles CURSORFORSELECT backupFile  FROM @fileList WHERE backupFile LIKE'%.trn'orderby idOPEN backupFiles FETCHNEXTFROM backupFiles INTO @backupFile  WHILE@@FETCH_STATUS = 0  BEGINif(len(@StandbyFilePath) > 0)beginSET @Query = 'RESTORE LOG ' + @dbName + ' FROM DISK = '''+ 
                        @BackupFolderPath+@backupFile + ''' WITH Standby=' + char(39) + 
                        @StandbyFilePath + @DBName+'.stdby' + char(39)endif(@StandbyFilePath isnull)beginSET @Query = 'RESTORE LOG ' + @dbName + ' FROM DISK = '''
                    + @BackupFolderPath+@backupFile + ''' WITH norecovery'endprint @Queryexec sp_executesql @QueryFETCHNEXTFROM backupFiles INTO @backupFile  ENDCLOSE backupFiles  DEALLOCATE backupFiles  End
/*
USP_Restore_Logs 
    @BackupFolderPath = 'C:\ToTestAndDelete\Logs\', --Last slash is needed
    @StandbyFilePath = 'C:\ToTestAndDelete\Logs\', -- Last slash is needed
    @BackupfileNamePattern = 'DBTRN_*', --File pattern
    @DBName = 'Test'
*/

This works fine in situations where TLog backup files follow same naming pattern. And since files restored not by name order but by Time order, restoration works.

Enhancing this stored procedure to

• Withstand different naming patterns (Small tweak of above procedure would solve it)
• Backup files may not be in one single folder but may be from multiple folders. So allow users to pass array of paths
• Instead of relying of Date for restoration order, restore by looking at header of Backup file and use First LSN and Last LSN to generate sequence of transaction logs to restore.

Coming to last point, there was a interesting problem that was found. The problem was because of system date changes by administrator, transaction log files lost name order as well as time order. Coming back to our TLog backup files example, see below they are not following Name Order, Date Order.

Currently procedure does not withstand such deviations. So we had to manually resort to putting them in time order or name order. Since SP executes by Time order we wanted to change Modified Date of each transaction log file. But before doing that we had to find out what is order of files that needs to be restored. To sequence out files we used trusted method of First LSN and Last LSN of TLog backup files.

To read First LSN and Last LSN of TLog backup files use

Restore headeronly fromdisk = '<FileNameAlongWithPath>'

We used simple code to dump all headeronly information for all TLog files into table and copied it to Excel and did our sequencing exercise (I am planning to do that in code and enhance stored procedure). In Excel we used color coding to find out sequence.

• FirstLSN is first LSN of Backup file.
• LastLSN is last LSN of backup file.
• Sequence if order of restore.

The last LSN of TLog backup should match with FirstLSN of subsequent file. If it matches that file is next file and if there are any gaps in LSN then it is LSN mismatches and logs hence forward can not be restored. Based on this logic we had sequenced all files for restoration. But how do we change modified date for each of such files. Remember some files had Modified dates at 1/1/2001 and other 1/2/2013 etc.. and it was quite random.

To modify “Modified File Dates” we used a tool called “File Date Changer” FileDate Changer

With this we modified dates in same sequence order and we are back in business….

As said earlier, enhancements for this SP

• Withstand different naming patterns (Small tweak of above procedure would solve it)
• Backup files may not be in one single folder but may be from multiple folders. So allow users to pass array of paths
• Instead of relying of Name or Date for restoration order, restore by looking at header of Backup file and use First LSN and Last LSN to generate sequence of transaction logs to restore.

Will enhance this procedure and share it, till then if there is a need use above SP.

If you need further enhancements to this procedure, let me know through comments sections or drop me an email at gurucb(at)hotmail(dot)com.

Thanks for reading this post and keep in touch.

Guru