Data Recovery

Background

My LG G3 smartphone suffered a bit of an ‘event’ recently which left several hundred pictures on the SD card inaccessible. A check of the card revealed that all the affected files were showing a size of zero bytes. This document outlines the recovery process and lessons learned. It also serves (IMHO) to demonstrate firstly, the power of even the most basic Linux system and secondly, that data loss can happen at any time and regular backups are essential! If I’d had a recent backup, none of this messing around would have been necessary.

First Steps

The most important thing in this sort of scenario is to stop using the affected storage device immediately. If any further data is written to the device from the point where the problem is first noticed, the chances of recovering any lost data will start to diminish quite quickly. With this in mind, the phone was shut down cleanly and the memory card was removed.

A byte-for-byte image of the card was taken using the dd command. A further copy of the image was then made – this is an added precaution in case the recovery process causes any further damage to the original image.

dd if=/dev/sdb of=card.img

cp card.img backup.img

Examining the Data

If you’re dealing with (for example) an ISO image of a DVD under Linux, you can usually mount it using the loopback device:

mount -t iso9660 some-image.iso /mnt/diskimage -o loop

The card.img file that has been written contains a copy of the whole memory card, hopefully including the partition table and both used and unused sectors of the file system, assuming neither have also been damaged. If I’d been sure that there was no damage to the partition table or the underlying filesystem, it would have been feasible to just take a copy of the partition I was interested in. I wasn’t 100% confident of what state the card was in, so I decided to play safe and take a copy of the whole thing. This means that I couldn’t just mount the image file using a simple loopback, I needed to be able to specify a particular ‘partition’ within the image file.

The Linux fdisk utility will display the layout of the card image data:

fdisk -lu card.img

Disk card.img: 7.4 GiB, 7969177600 bytes, 15564800 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0x00000000

Device Boot Start  End  Sectors  Size Id Type

card.img1    8192 15564799 15556608  7.4G  b W95 FAT32

The important things to note are the ‘Units’, which are here reported as sectors of 512 bytes each, and the start sector number of the first partition, in this case 8192.

We know from the fdisk output that each sector is 512 bytes and that the filesystem starts at sector number 8192. Multiplying these two numbers together gives a value of 4,194,304 – this is how far after the beginning of the card image data the filesystem actually starts at. You can pass this number as an option to the losteup command and then mount the filesystem as normal:

sudo losetup /dev/loop0 card.img -o 4194304

 sudo file -s /dev/loop0

 /dev/loop0: DOS/MBR boot sector, code offset 0x0+2, OEM-ID "    ", sectors/cluster 64,    reserved sectors 4394, Media descriptor 0xf8, sectors/track 63, heads 255, hidden sectors    8192, sectors 15556608 (volumes > 32 MB) , FAT (32 bit), sectors/FAT 1899, serial number   0x6a5661fd, unlabeled

 sudo mkdir /mnt/cardimage

 sudo mount /dev/loop0 /mnt/cardimage

 sudo ls -l /mnt/cardimage

 total 640

 drwxr-xr-x  2 root root 32768 Jan  2  2014 Albums

 drwxr-xr-x  5 root root 32768 Jul 25  2016 Android

 -rwxr-xr-x  1 root root  3262 May 25  2011 android.ico

 drwxr-xr-x  3 root root 32768 Sep  8  2016 data

 drwxr-xr-x  5 root root 32768 Mar 26 12:45 DCIM

 drwxr-xr-x 12 root root 32768 Aug 22  2014 eBooks

 drwxr-xr-x  2 root root 32768 Oct  2  2012 LOST.DIR

 drwxr-xr-x  3 root root 32768 Jan  2  2014 media

 drwxr-xr-x 23 root root 32768 Jan  2  2014 Music

 drwxr-xr-x  2 root root 32768 Mar 24  2015 My Documents

 drwxr-xr-x  2 root root 32768 Dec 19  2012 Nearby

 drwxr-xr-x  2 root root 32768 Jul 25  2016 Notifications

 drwxr-xr-x  2 root root 32768 Sep  1  2014 PhotoEditor

 drwxr-xr-x  2 root root 32768 Jan  2  2014 Ringtones

 drwxr-xr-x  2 root root 32768 May 24  2014 Sounds

 drwxr-xr-x  2 root root 32768 Jan  2  2014 svox

 drwxr-xr-x  2 root root 32768 Mar 26 13:04 System Volume Information

 drwxr-xr-x  2 root root 32768 Aug 21  2014 Videos

 drwxr-xr-x  3 root root 32768 Aug 23  2014 VoiceRecorder

 -rwxr-xr-x  1 root root   296 Oct 25  2012 WMPInfo.xml

The filesystem seems to mount okay, yay! But the problem with the corrupted images can be clearly seen, boo!

sudo ls -l /mnt/cardimage/DCIM/Camera/|less

total 2278336

-rwxr-xr-x 1 root root 448582 Jul 16  2005 20050716_181123.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20050807_161821.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20060310_180404.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20060407_141730.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20060407_141751.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20070412_154555.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20070527_133128.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20070527_133953.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20070602_130125.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20071004_184144.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20080318_040212.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20080506_143219.jpg

-rwxr-xr-x 1 root root     0 Mar 26 13:34 20080513_124327.jpg

You can now unmount the image and reset the loopback device /dev/loop0:

sudo umount /mnt/cardimage

sudo losetup -d /dev/loop0

Initial Data Recovery

The software used to attempt recovery of the missing data was part of a package called Testdisk, see the references section for the URL. This comes as a statically-linked binary – this means it has no dependencies on other libraries that may or may not be installed on your machine. In other words, it will ‘just work’!

You need to specify the source of the data that it will try to recover, in this case the image of the memory card, as well as the output directory where recovered data will be stored. An option is also added which caused a log file to be created, this may be useful later in the process.

./photorec_static -log -d ./recovery card.img

The software uses a fairly straightforward curses-syle interface. The saved card image is already selected, as per the command line options:

The next step is to choose the partition within the image:

Now confirm that it has correctly identified the filesystem type

Finally, select the option to scan the entire partition for data. This takes longer, but will improve the chances of recovering the corrupted images:

Now you just have to wait while the software does its thing.

Analysis – Recovered Images

The next step was to see how many images photorec had apparently recovered.

   find . -type f -name "*.jpg"|wc -l

    2760

Hmmm….I definitely didn’t have that many pictures on my phone. I used the identify utility, which is part of the ImageMagick toolset, to try to get a better insight into what’s been recovered. Here’s a small section of the output:

   find . -type f -name "*.jpg"|xargs identify

    ./recovery.4/t14502528.jpg[2741] JPEG 512x288 512x288+0+0 8-bit sRGB 49.1KB 0.000u 0:00.009

    ./recovery.4/t14534528.jpg[2742] JPEG 320x240 320x240+0+0 8-bit sRGB 8.09KB 0.000u 0:00.000

    ./recovery.4/t12924992.jpg[2743] JPEG 320x240 320x240+0+0 8-bit sRGB 20.5KB 0.000u 0:00.000

    ./recovery.4/f13214272.jpg[2744] JPEG 4160x3120 4160x3120+0+0 8-bit sRGB 3.429MB 0.000u   0:00.000

    ./recovery.4/t13032896.jpg[2745] JPEG 320x240 320x240+0+0 8-bit sRGB 10.2KB 0.000u 0:00.000

    ./recovery.4/t13077632.jpg[2746] JPEG 320x240 320x240+0+0 8-bit sRGB 16.1KB 0.000u 0:00.000

    ./recovery.4/f13329408.jpg[2747] JPEG 4160x3120 4160x3120+0+0 8-bit sRGB 3.993MB 0.000u   0:00.000

    ./recovery.4/f14321728.jpg[2748] JPEG 4160x3120 4160x3120+0+0 8-bit sRGB 3.766MB 0.000u   0:00.000

    ./recovery.4/f14534528.jpg[2749] JPEG 4160x3120 4160x3120+0+0 8-bit sRGB 2.449MB 0.000u   0:00.000

    ./recovery.4/t14982272.jpg[2750] JPEG 320x240 320x240+0+0 8-bit sRGB 9.1KB 0.000u 0:00.000

    ./recovery.4/t12988608.jpg[2751] JPEG 320x240 320x240+0+0 8-bit sRGB 22.2KB 0.000u 0:00.000

From this, I could see that the recovered images had many different sizes. Some looked like they might be the original images that were lost, for example the ones with dimensions of 4160x3120. Others looked like they might be just thumbnails or cached images generated by the apps I have installed, based on the dimensions being much smaller, for example 512x288.

Pictures taken by the camera app on my phone will have EXIF data embedded in them, which identifies (amongst other things) when the picture was taken, what device was used to take the picture, what exposure settings were used, etc, etc. The jhead utility can be used to check for the presence of EXIF data in an image, so let’s see what happens if I try it on the 2 recovered images files highlighted in the listing above:

   jhead ./recovery.4/t13077632.jpg

    File name : ./recovery.4/t13077632.jpg

    File size : 16085 bytes

    File date : 2015:10:31 21:27:01

    Resolution   : 320 x 240

    JPEG Quality : 75

   jhead ./recovery.4/f13329408.jpg

    File name : ./recovery.4/f13329408.jpg

    File size : 3993427 bytes

    File date : 2015:11:10 19:00:50

    Camera make  : LG Electronics

    Camera model : LG-D855

    Date/Time : 2015:11:10 19:00:50

    Resolution   : 4160 x 3120

    Flash used   : No

    Focal length :  4.0mm

    Exposure time: 0.091 s  (1/11)

    Aperture : f/2.4

    ISO equiv.   : 400

    Whitebalance : Auto

    Metering Mode: center weight

    GPS Latitude : ? ?

    GPS Longitude: ? ?

    GPS Altitude :  0.00m

    JPEG Quality : 97

This looks promising. The smaller sized files don’t seem to have any EXIF data in them at all, whereas the larger recovered files appear to have the complete info. It’s time to start gluing some of these bits together into a script that I can use to identify exactly what I’ve got and pull out the images that I want to keep.

Scripting

As a first step, let’s put together something that can identify whether an image file has EXIF data or not, simply by checking for the presence of the ‘Camera model’ string and outputting either the string that was found, or the word ‘None’:

    vim script1.sh

   #!/bin/bash

    # Check for EXIF 'Camera model' string in image file

    # make sure the command line includes a parameter

    if [ $# -eq 0 ]; then

    echo -e "ERROR: missing command line parameter\n"

    exit 0

    fi

    # see if the parameter is a valid file name

    if [ -f $1 ]; then

    MODEL=`jhead $1|grep "Camera model"|cut -f2 -d":"|cut -c2-9999`

    if [ "x$MODEL" == "x" ]; then

        echo -e "$1 :None"

    else

        echo -e "$1 :$MODEL"

    fi

    else

    echo "ERROR: file $1 not found\n"

    exit 0

    fi

Remember to do a chmod 755 script1.sh to make it executable. Running this against the 2 example files highlighted above gives these results:

    ./script1.sh ./recovery.4/t13077632.jpg

    ./recovery.4/t13077632.jpg :None

    ./script1.sh ./recovery.4/f13329408.jpg

    ./recovery.4/f13329408.jpg :LG-D855

So now let’s run this against every single recovered image file and see what happens:

   for i in `find . -type f -name "*.jpg"`; do ./script1.sh $i >> script1.out; done

To save space, I won’t paste the output generated by checking all 2,760 recovered files. Examining the output file, however, showed that there were a lot of different ‘Camera model’ strings in the images. This will be because I’ve got (or rather, had!) pictures stored on the memory card that I’d originally taken with phones that I’ve long since gotten rid of and/or upgraded, or pictures that have come from other sources. A quick bit of grep/sort magic, ignoring the files which don’t have any EXIF data, reveals exactly how many images I’ve accumulated over the years from each device:

   cat script1.out|grep -v None|cut -f2 -d":"|sort|uniq -c|sort -n

     1 GT-I9100

      1 KS360

      1 LG-H850

      2 KU990

      2 N8-00

      3 FinePix S7000

      5 FinePix A340  

      5 N93

      8 6680

    23 N95 8GB

    27 HTC Desire HD A9191

    99 N900

    110 Desire HD

    144 GT-I9300

    953 LG-D855

It’s like a little potted history of my family’s mobile phone and digital camera ownership! While we’re here, let’s just check how many images have been recovered which apparently have valid EXIF data:

   cat script1.out|grep -v None|wc -l

    1384

That looks like a slightly more reasonable number. It’s worth bearing in mind that the recovery process may also make previously deleted pictures re-appear, so there will be some housekeeping required later on.

The next thing I need is a script that will take any image with a valid EXIF ‘Camera model’ tag and move it to a folder which is seperate from the raw recovered data.

   vim script2.sh

    #!/bin/bash

    # Move a recovered image away from the raw recovered data

    # make sure the command line includes a parameter

    # and that it's a valid file name

    if [ $# -eq 0 ]; then

    echo -e "ERROR: missing command line parameter\n"

    exit 0

    fi

    if [ ! -f $1 ]; then

    echo -e "ERROR: file $1 not found\n"

    exit 0

    fi

    # get the 'Camera model' tag using script1.sh

    # make sure it isn't 'None'

    MODEL=`./script1.sh $1|cut -f2 -d":"`

    if [ "$MODEL" == "None" ]; then

    echo -e "ERROR: invalid camera model string in $1\n"

    exit 0

    fi

    # rename the recovered file based on the EXIF data

    NEWNAME=`/usr/bin/jhead -n%Y%m%d_%H%M%S $1|cut -f3 -d" "`

    DESTFILE=/home/phile/keepers/`basename $NEWNAME

    # move the renamed file into the keepers folder, but let's

    # be picky and not overwrite if the destination already

    # exists

    if [ -f $DESTFILE ]; then

    echo -e "ERROR: destination $DESTFILE already exists\n"

    exit 0

    fi

    /bin/mv $NEWNAME $DESTFILE

    # set the file timestamp based on the EXIF info

    /usr/bin/jhead -ft $DESTFILE > /dev/null

    # rename the copied file based on the EXIF info

    echo -e "$1 renamed to $DESTFILE"

Don’t forget to do a chmod 755 script2.sh to make it executable. You will also need to change the ‘DESTFILE=’ line (highlighted above) so that it points to your home directory. To run this on all the recovered files:

   for i in `find . -type f -name "*.jpg"`; do ./script2.sh $i; done

You should end up with copies of pretty much all of your recovered images!

References

[1] Loop mounting specific partitions from a disk image, http://madduck.net/blog/2006.10.20:loop-mounting-partitions-from-a-disk-image/

[2] Testdisk, data recovery, http://www.cgsecurity.org/wiki/TestDisk

[3] ImageMagick, Convert, edit or compose bitmap images, https://www.imagemagick.org/script/index.php

[4] jhead, EXIF jpeg header manipulation tool, http://www.sentex.net/~mwandel/jhead/