Re: [Nolug] RHEL6 filesystem on lvm recovery

On 10/30/12, j j <jdj2005@gmail.com> wrote:
> Here is one for the hard core/black belt administrators out there.
> I have a blade running RHEL6 connected to a san. The storage is managed by
> LVMs. Of course we had a meltdown because the SAN administrator decided to
> make a change on the san, without notification, and basically trashed the
> LVM superblock on the associated LUN and has no backups of the LUNS. The

Dare I ask, what change exactly.... Assuming the SAN isn't just
itself defective in some particular way (How sure can you be it won't
happen again?); it sounds like some horrible atrocity was committed
against the volume. a SAN's not supposed to lose or corrupt data on
an existing LUN, unless an initiator connecting to a target mapped to
the LUN, and granted write access, has written bad data to the sector,
or the SAN failed to write the good data to the right place....

The idea that you just lost all the LVM bits, and somehow all the
data you want is intact; well _maybe_; if there was just a narrow
region of the volume that got junk written to it.

If it's just a few sectors at the start of the LUN, it might be
something you can work out.
If you are quite sure the damage is limited to LVM metadata, then
there are sure to be some recovery options; they just may take a lot
of time, and potentially expense to implement; specifically: looking
for a needle in a haystack... ASCII contents of /etc/lvm/archive,
for PV restoration.

Step 1 should probably be DD'ing the entire contents of the disk
that every LVM PV was on to offline media, and the disk containing
the /etc/lvm (if /etc is not on the LVM).

Don't start flipping any bits on a seriously damaged PV/VG/LV or
filesystem, without a backup; even if the backup is also
unmountable, it's possible the data is recoverable but an initial
recovery attempt will go horribly wrong.

With read-only image copies in hand, then see about reconstructing LVM
 physical volume metadata to the original media.

Only with valid metadata for all PVs in the VG, will it be possible to
activate the VG normally, via the vgchange command.

> LVMs can be reconized by RH but the filesystems cannot be mounted or even
> seen because there are no superblocks available. fsck and vgconfigure do
> not work, again because there are no superblocks. Data can be seen with a
> dd dump.

LVM superblocks, or Ext3 / Ext4's superblocks inside the FS? All
Superblocks, including the backup superblocks of each filesystem are
corrupted?

If you actually still see the VG and LV but all the filesystem
superblocks everywhere have been broken; then that's an extensive
inconsistency, that will likely mean the filesystem cannot be
adequately repaired in place. Probably game over.

But some commercial Ext3 recovery tools
may still be able to extract some files in that situation

--
-JH
___________________
Nolug mailing list
nolug@nolug.org