Hi
I am running a Lotus Domino 8.5 server on Centos 5.2 x64 Linux. The installation is on a Intel server with Adaptec RAID controller 1.7Tb and the Notesdata directory is on an external Dell PowerVault MD1000 5.5Tb with RAID 5 configuration. I also have LVM2 installed.
On Wednesday last week we had a power surge. I have 2 x 3KVA UPS installed, but for some reason the external Powervault RAID lost the connection to the hard drives. After i rebooted, i check the RAID BIOS and found that 1 drive failed and the RAID is rebuilding.
After the process completed, i started up the server. FSCK gave me error that the volume is inconsistent and i should run fsck manually. So i ran fsck -y /dev/VolGroup02-LogVol02
This ran for almost 10hours, and supposedly fixed hundreds of errors. Once the server booted up, the file system was mounted as read only. I tried unmounting and remounting the volume, also checked fstab for valid mount points.
fdisk -l stated that the volume does not contain a valid partition table. So i bought 3 x 1tb drives and started transferring the data. I cleared and re-initialized the RAID and created a new RAID5 config.
This time i created a GPT partition table and formatted the file system to ext3, transferred the data back using Krusader and preserved ownership and permissions.
I ran an offline fixup, compact and updall.
I then fired up the Domino server to find that my names.nsf is gone. So copied one from a backup server. On startup i found many errors about files that do not exist. I copied the missing files from the backup server and all went well.
Only problem i am stuck with is user mail files. Some files when opening will give error "File is not a database". The mail files have sizes ranging from 260Mb to 4Gb.
I tried running a fixup, but got the same error that the file is not a database. I have replicas of the user mail files on my backup server but there are a couple of mails missing and i am unable to replicate databases hence the error.
What i have done for the time being is rename the problematic mail files and pushed new replicas from the backup server. The result is a loss of mail and folders for dates 12 - 22 March.
What can i do to recover/repair these mail files?
Maybe you can find useful information on the website of IBM in this article?
http://www-01.ibm.com/support/docview.wss?uid=swg21085746
The BIG problem with UNIX/Linux file-system repairs is that blocks of data of one file could (theoretically) end up in another, and file size corrections can lead to blocks of empty data or truncated files. This is not a real problem for plain text files, since you can see what's wrong, but binaries and files with complex structures (like nsf files) can get damaged beyond repair. And if fsck runs for 10 whole hours, there must have been some serious damage, with a very lengthy report of all modifications it did.
I'm amazed that the system still runs... See it as a boxer who's lights went seriously out, was patched up and is now covered in patches and stitches. On the other hand, the system will run because it's unlikely that there was any write-behind action on vital system files.
You might ask IBM/Lotus for their help, maybe they have a tool that can be used to revive an nsf database... Or maybe it's better you try to recover a file via Lotus Notes Repair Kit tool? try this option also. I'm sure this is a good alternative fallback. http://www.lotusnotes.repair/
Look at this reputable source also.
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Database_Corruption_Troubleshooting_Guide
Good Luck.
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.