Data (image) loss .... stranger things!

[arthurking83]

:confused:

Never seen anything like this before!
I've had data loss, couple of times my fault, most times PC fault, or memory card issues.
But this one caught me totally off guard, and still has me stumped.

So I got a few images, can't remember exactly how many now, but my usual fool proof routine is:
* Card into ext card reader.
* Transfer using Nikon transfer.
* Opens ViewNX2 automagically to review images.
* select some that have some potential, edit in CNX2

With all that done, I saved my first image(the locks on the gate image .. HERE (http://www.ausphotography.net.au/forum/showthread.php?159662-Insecurity!))
So this image is the original version pre crash/loss.
It's original file name was DSC_2989.

In some of my efforts, I took a series of snaps, to create a panorama.
So, in ViewNX2 again, select two images to pano, convert to tiff, open in PTGui.
All good so far.
in doing that, I still had VNX2 open, and displaying one of the other(non pano) images .. some yellow flowers, I think one of the last images of the day, so wayyy over to the end of the film strip(in VNX2).

Had PTGui make the pano creation thing, worked out OK, then back in VNX2(which was minimised) to see the final PTGui tiff file it saved.

This is went wrong!
ViewNX2 was now in a totally different directory area(My Pictures), not where I left it minimised in the new images for that session.
Strange! It only does this if you move the images it's showing using another application. It gets confused as it no longer sees the image it was supposed to be displaying .. and defaults. Default is to go back to the My Pictures folder.

Anyhow, didn't think anything of it, just tried to navigate back to my new images .. except they were not there! .. all gone. Not just one or two, all gone.
Made no sense at all.
The locks images saved before was there, the Pano folder with the tiff files, two images and the PTGui saved tiff file were in there too ... just all the raw files in the folder I created for that session were gone.
First thing I thought was that I may have accidentally deleted them .. but I didn't delete anything at any time.
Checked recycle bin .. nup! lot of old files from wayy back ... but not these new raw files.
Getting stranger!
Used Recuvva to scan the specific directory I transferred the raw files into... found lots of other files around the directory, none spcifically within.
Getting even stranger than stranger!
It was like I'd never transferred them.
Used Easeus Recover .. same deal as above .. which I knew would be.

So back to the CF card. I have Nikon Transfer to delete the images after transfer, so Recuvva used on the CF card now.
Found all of them, but they were differently named to how I name them via the camera. I use DSC, DSD, DSE .. etc + plus the cameras natural numbering system as a semi chronological method of file naming my files.
The Recuvva recovered files all had SD_file number, and the numbers were all off by a about 10.

So it gets even stranger again. Recuvva shows you the expected possibility of recovery for each file. It showed two or so unrecoverable, most excellent.
Did what I could, and it recovered the unrecoverable ones.
But it didn't recover all the excellent probabilities! :confused013
About the last 10 or so, it missed.

So, back to Easus recovery: it showed all files, but gives no idea of the probability of success, it just does it's thing.
Hit the go button, and now .. it gets stranger than the original stranger than strange madness .. Easus recovers the files that Recuvva showed as excellent, but couldn't recover!
But Easeus didn't recover the files that Recuvva did!

Never seen anything like this. I never wrote to, or deleted anything off the SSD(where all the images were transferred too) during this twilight zone timeframe.
Couldn't understand why one recovery program couldn't recover some images, and the other could, and vice versa.
Lost about 3-5 images .. with the new file naming could be sure how many exactly .. not worried as I wasn't overly enthused by any.

And it 'corrupted' one image, as a raw file. The locks on the gate image in the link above.
VNX2 has the ability to batch file rename, so I used it to rename the files(from SD_... ) to my preferred D800E_DSC...
all good, except for that one corrupted one.
On initial viewing of the recovered files, VNX2 displayed it as per normal. CNX2 and CNX-D displayed it corrupted.
VNX2 generates jpg preview files from the embedded preview file within the raw file. So whilst the raw file may be corrupted, initially, the embedded jpg file wasn't.
Once I made a change to the corrupted file in VNX2, it then re rendered the embedded preview file to display the corruption now. No going back, but I kind'a like the corrupted version now.

So in the above link, I show the converted original image before all the stranger things occurred.
Below I show the now corrupted version:

http://www.ausphotography.net.au/forum/dbtgallery.php?do=gallery_image&id=3846&gal=gallery&type=full
As can be seen, there are three images in this one. No idea why the fuchsia coloured area tho .. that's wayyy out there corruption.
But the thin RHS shows two sections of an image. They all appear to be the same image, just jumbled up a little.
Having looked at it, I think the two RHS sections are reversed, that is the pinky bit should be above, the the normal coloured part below, and they are on the correct side of the image .. just separated.

[ameerat42]

If you had made it up it wouldn't be as strange.

Take it easus... and have a good lie down until you recuvva...:cool:

I guess you'll be making some changes to your routine :eek:

[Bianca]

That sounds really weird. I wonder if you accidentally moved your session folder on the SSD elsewhere not immediately visible (into a subdirectory perhaps)? Not sure which OS you're using but if Windows, I know that Windows Explorer is deplorable and can't seem to find anything, which is why I use Agent Ransack (free) to search.

Doesn't explain your weirdness on the CF card, however!

[arthurking83]

....

I guess you'll be making some changes to your routine :eek:

Not soon!
Been solid as a rock for many years now(until this faux pas).

Like i said tho, it wasn't as tho I'd done something(like delete or something).
Had I, or had they deleted some other way, any recovery program will show them as having been there, with varying degrees of recoverability.
But that's where the stranger things element came into it .. nothing there.

I did a whole drive search, to be sure, maybe I didn't transfer them into that specific directory, and doing that, discovered many files deleted wayy back in 2017 or so .. so I don't question the recovery softwares.

Only possible explanation is that the SSD had a hissy fit, or some type, and dumped those missing files.
However, that also doesn't make sense, as I'd have thought it'd also remove the converted jpg, and tifs created prior to the loss as well.

Have thought to tweak my routine, in that Nikon Transfer does have the option to transfer the files off card(or camera, or whatever) to a backup location.
Issue is a few fold with that, mainly speed. I transfer to a SSD, and each 90Meg file takes hardly a sec, so transfer is quick. Having only non SSD drives for backup, will make the process slower.
Also, I create specific folders to download too at the time of transfer. So it's tedious to repeat that process for the backup transfer part. Otherwise, I'd set a specific backup location and have to delete them every now and then .. and being of the lazy variety of human .. that set backup location could easily end up over 4Tb before I remembered to clear it!

For my actual backup routine, I use FreeFileSync. easy to use, but I do use a slightly tedious method .. lesson learned the hard way.
I manually check almost every listing to be sure stuff I don't want backed up doesn't get backed up .. unless I really REALLY! do want to re back it up.
Method is a simple Page Down/Up key scrolling, watching the file listing fly by. I just watch for 0.001 sec blips of files whizz by, and then scroll back to see what they were, and then either do the backup, or set those to "do nothing" when the actual sync button is pressed.

Got caught out once where I backed up corrupted data over good data(unknowingly of course), by using an almost fully automated backup routine .. wont make that mistake again!

I will do a few system checks on the SSD soon too. My suspicion is that this is what went goofy.

Every year, I do a clean out of the old(ie last years) files .. delete them.
This SSD is just a holding drive, due to it's speed. Only holds a year or so raw files(when I used to do more).
My first backup is to a 4Tb HDD, which I do regularly .. but not same hour .. maybe every month, or two.
This isn't my primary concern as backup.
My real backup is to a NAS, where I try to be as 100% sure those files are backed up, but that I'm not backing up any corrupted raw files.
I have two drives on there for data safe keep that I sync every so often.

[tandeejay]

I have seen 4 occasions in my life when strange things like this have happened.

Twice occurred while I was at uni nearly 30 years ago when I changed floppy disks while a particular application was running, the root directory from the original directory got written to the 2nd disk, so the 2nd disk got completely corrupted. The 2nd time it occured, I was able to use something like norton utilities to re-construct the directory by examining the file chains in the file allocation table, and rebuilding a root directory to point at all the recognisable file chains. Only lost about 10 minutes of work on an assignment... but it was text documents, and floppy disks only stored 360k of data, so was a feasible recovery operation...

The 3rd time it happened, I was using Motherboard RAID 1, and I was doing a windows upgrade, so I thought I'd be "smart" and unplug one of the mirror pairs so I'd have an easy rollback point. When I plugged the drive back in, the motherboard saw the disks as part of the same mirror pair and tried to re-synchronise them, only it didn't do the synchronisation one way, it just randomly picked data from one disk and wrote it to the other disk, thus corrupting both disks...

4th time occurred on a server in a datacenter... a cache battery in a raid controller had failed some 6 months previously, and the raid controller had disabled the cache... then when I discovered the failure, and raised a support call, the backend engineers stated "the server and disk array must be shutdown before replacing the battery", then the field engineer, when he came on site, said "No, the battery can be replaced hot"... the field engineer then went against the backend engineers recommendation and replaced the battery with the system still running... the raid controller then decided that the battery was good, and it could start using the cache again... only it didn't assume the cache was empty but started using the cache with it's 6 month old data, thus promptly corrupting the filesystem... (I then had to spend the weekend recovering that filesystem from backups...)

So, from your description, it sounds like something has written to the filesystem a cached (old) version of the directory in conflict with what it should be. Are you running any sort of caching software/hardware/ "speed up your PC" type software, or do you have a raid controller that has battery backed cache? Could also be your SSD has a failure in it. You don't defragment your SSD do you.

I think I'd be wanting to run a full filesystem integrity check to make sure there is no other corruption anywhere else on that disk. and maybe even check images in other folders to make sure the corruption is not more widespread. Also, make sure your malware/virus scanners are up to date and run a full system scan to ensure you have no malware.

and even take a full system backup separate from your existing backup strategy, in case you need to do a restore from your existing backups. With a backup of the system as it is now, you can restore your last known good backup, and then have the current backup to potentially restore anything that was added since last known good backup... but be careful that you don't overwrite your good data with corrupted data that might be in the good backup.

[Tannin]

I read it.

Doctor, my brain hurtz!

[mudman]

your image reminds me of this image i posted 2 years ago
i noticed the other day the chain has been changed to one with only 1 lock.
that's progress for you

138232

[Colin B]

My brain hurts too and I am a Nikon user.

My routine, for what it is worth is not to use Nikon's software at all - I find it clunky and difficult. I stick the SD card into the computer, open it and plonk all the images into a file labeled "today's download" in my "camera images" file. I them eject the card and set it aside without deleting anything except, perhaps, a few hopeless pics.

Any post work I do on the copies in the computer before placing the keepers into their various files for permanent storage. Only when I have finished all my fiddling and the new images have been uploaded to Dropbox do I put the card back into the camera and format it. That way if I (usually) or the computer stuffs up the image is not lost.

Don't ask how I learned to do it this way- it is embarrassing.

[arthurking83]

.... You don't defragment your SSD do you.

.....
Nup!
no defrag, file system checks out.
It's a Kingston branded SSD, and I installed the utility prog that comes with it, nothing in the logs to indicate any anomaly, all health checks come up good.

Problem with this now is that I'm more worried about it. Maybe SSD on it's way out, and this may be a way of indicating this .. but the branded software may be hiding the truth(or some other conspiracy theory to that effect).

I'm not fully sold on the old cache theory tho:
I'd have thought if this was the case, the long since deleted files(about 1800 or so) jpgs and small text files with a .map extension would have all been destroyed as well(in terms of deleted files).
They were all from early '17.
The fact that they were all there and supposedly excellent chance of recovery points to a healthy file system.

I don't do a lot of activity on this SSD drive, other than the one folder I use as another cache(for speedier software operation).
The normal routine is dump 'todays' photos on there to a specific folder.
Every year or so, they end up getting deleted as this photo store is purely for speedier operation of the current set of images. Nikon software works a lot faster on fast storage space .. hence this dedicated SSD.
One of the other things I do a fair bit is update/make maps for my tablet GPS unit.
This is a kind of panorama making program, where lots of small map images are stitched into large map images(which I then use on my GPS).
Faster storage is recommended where the stitching operation occurs.
So I've made a folder called cache on this SSD.
my every day use cache (ie. for almost all other programs) is on a normal HDD.
I know that SSDs have a supposedly limited write cycle compared to a magnetic drive(all else being equal).
That is, there is a chance that writing-deleting-writing stuff to a SSD too many times can cause those memory slots to be exhausted.
So I specifically try to limit how many times I rewrite to the SSD.
Oh! and I just checked, and I did format this SSD, along with most of my drives in GPT format too. Supposedly less susceptible to data loss! :rolleyes:



I read it.

Doctor, my brain hurtz!

Don't blame 'ya.


My brain hurts too and I am a Nikon user.

My routine, for what it is worth is not to use Nikon's software at all - I find it clunky and difficult. ....

I have no issue with Nikon software.
I don't find it clunky at all .. in fact a bit too simple compared to some others, but does what I need/require.
But we all have different needs, and my main one is to maintain all additional descriptive data, that I invariably add, within the raw files and only Nikon software does .. did!.. this properly.

I'm 99.9% sure it wasn't Nikon software fault, and 99.9% sure it was a hardware fault.
if it were software fault, the files would have been discovered as deleted by the recovery programs used.
Not being discoverable .. (in my guesstimation) points to a hardware fault.

Thinking about this more, if I do have an impending hardware issue, I may take on Am's advice and re tweak my workflow and use that annoying backup feature in Nikon Transfer.
I may also set Transfer to not delete the files off the CF card too, just in case.
I can easily format the card in camera when re installed.

[jim]

This is all deeply confusing so I'll make no comment on Arthur's predicament, let alone John's incomprehensible, yet I assume meaningful, reply. I'll just note that when I transfer files to my computer I copy and paste to the appropriate folder, which avoids messing with Nikon's not always superb software, and leaves everything intact on the card until I decide that the operation has concluded successfully and I can go ahead and format it.

now, where's that "smug" smilie?

[tandeejay]

My previous coment were just my thoughts on possible causes, and not a firm diagnosis.

I just had another thought. How close to capacity is the SSD? Just wondering if your usage happened to exceed the TRIM operations ability to free up blocks in the SSD. When you write changes to files on an SSD, the old version of the file continues to take up space until the OS triggers the TRIM operation. I've never seen what happens in that scenario so not sure if your symptoms point to that happening. I'm not sure what frequency Windows the TRIM operation. I know on linux you can set filesystems to TRIM immediately or you can set it on a schedule, but there are warnings that setting it on immediate can have a performance impact on certian workloads. Did any events appear in the event viewer at the time of the issue?

Sent from my LG-M700 using Tapatalk

[tandeejay]

let alone John's incomprehensible, yet I assume meaningful, reply.

Hahaha, sorry, I've been in the Computer industry since the late 80's and if I forget my audience, I can get too technical... When people ask me what I do for a living I usually just say "I look after computers" then i usually have to qualify that with "the really big ones not desktops and laptops" or they start asking me about their own computer problems :D


Sent from my LG-M700 using Tapatalk

[arthurking83]

..... I'll just note that when I transfer files to my computer I copy and paste to the appropriate folder, which avoids messing with Nikon's not always superb software, and leaves everything intact on the card until I decide that the operation has concluded successfully and I can go ahead and format it.

now, where's that "smug" smilie?

How many different camera models of the same brand do you have, or have had.
I used to do this too, and caught myself out majorly.
Don't know how other brand handle file names, but Nikon use DSC_1234 format by default.
Having gone through one camera, then another and again another and another again, invariably you get the same file names(eg. DSC_0001) at some point.
With the lower end, basic cameras, file naming in camera is extremely limited in that you don't have many options for renaming in camera.
So you go from DSC_9999 to DSC_0000 and _0001 again.
By placing your files in dedicated folders and not just all dumped into the one photo folder, it's easy to avoid overwriting old files with new files .. but it can still be done.
But more importantly with the multiple camera situation, its a lot easier to overwrite one cameras file with another from a different camera but with the same name.

You don't need to ask if I've ever done this ;) .. I do use folders to avoid this, but it was when placing some files collectively in another folder, some from the D70s and others from the D300 that I realised this was a bad situation.
I clocked my D300 about 7 times, but it allowed the ability to alter file names(D70s didn't) so I'd get to file DSC_9999 or thereabouts and change the file naming in camera to DSD_xxxx and DSE_xxxx .. etc to maintain chronological ordering.

Did it the copy/paste method for about the first year and a bit, and realised copy-pasting wasn't the ideal way to do it ... hence why I started using Nikon Transfer.
I use the file renaming feature and now when files are transferred, a prefix is added according to which camera they come off .. eg. D70s_DSC_0001, D300_DSC_0001, D800E_DSC_0001, or D5600_DSC_0001.

Much later, I realised my major mistake in not adding keywords as my archive grew to over 100K images. now 200+K .. but only because I've haven't had a lot of time to get out and do more .. I was shooting on average over 1000 image per week.
Try finding a specific image with a specific feature in it, with thousands upon thousands of images to sift through. as an example, of this issue:
When I was a courier, I'd do a Bendigo run almost every couple of days. To not stop and take the odd landscape, or detour via Maldon or whatever other excuse not to take a photo just made no sense.
So I have about 100+ folder of images in the Bendigo area. Trying to specifically pinpoint an image of a rundown/dilapidated bridge I knew I took in the area, but not when .. so finding stuff was becoming a nightmare.
So keywording/tagging images with appropriate data was becoming essential.
I didn't like the psuedo tagging system Adobe developed with their external xml method, I much prefer the embedded data system whereby the tag data is internal to the file. ie. basically much less can go wrong the simpler the system.
Easiest way to add tag data to a raw file is via the manufacturers software(if possible), as there is a lower likelyhood of file corruption. Can be done using other software, but I like to add some at transfer time.
The point isn't to add it all at the time of transfer, the idea is to add something to get those images into the loop(of having tagged data) which then makes it easier to add more to differentiate images from each other where required.



My previous coment were just my thoughts on possible causes, and not a firm diagnosis.

I just had another thought. How close to capacity is the SSD? Just wondering if your usage happened to exceed the TRIM operations ability to free up blocks in the SSD. ....

Comments always appreciated John .. wasn't having a go or anything like that.
Just the observation that the old files, over 2K of them, and about 1800 of those map files still there from 2 years ago.

Space no problem. 256G drive, never had more than about 30-40G at any one time. after a small cleanout(1G or so) it's now back at about 25-30G .. so tons of space.

Like I said, it's like they never existed on the drive .. ever. You'd think that a recovery program(or two) would locate some trace of them between the two.
I can find the recovered files(from the CF card) that were cached to this SSD drive using the recovery software. I then copy pasted them all to the specific folder that I originally placed them all into, then deleted the cached recovered files.
I just reopened the recovery software and they come up in the list, as well as all those old mapping files from wayy back .. and some other files from 2015, when I first built this desktop.
Just no listing for the lost files off that drive.
I initially thought, maybe I transferred them to another drive(I have 7, 2 of them primarily dedicated to photos storage) .. so checked that one, nothing.
I know they did get transferred, as the folder created for them is still there(FWIW: 20190106_Ararat_Ben Nevis), the converted jpg file from the first 20mins or so after the transfer in it's jpg sub folder is untouched.
I since renamed it(that's the lock photo) to suit the new name of the raw file of that image that got recovered.
Also, still had the tif files(3 of) from the PTGui pano effort, and they were deleted much later .. after all the mess was sorted.

I can't ever remember ever using a secure erase routine on any drive .. ever. So that's not a probable cause .. just weird that they seem to have never existed on the PC.

[jim]

Why do you retain the "DSC'? It provides no information and most likely all your cameras will let you change it to something more useful.

or do the older ones like the D70 not allow it? I know the D300 does, but I don't recall if the old D80 did. Still, no problem if you have only one camera that has to use DSC.

[arthurking83]

Why do you retain the "DSC'? It provides no information and most likely all your cameras will let you change it to something more useful.

or do the older ones like the D70 not allow it? I know the D300 does, but I don't recall if the old D80 did. Still, no problem if you have only one camera that has to use DSC.

D70s doesn't allow any alteration. Will only change the DSC component to _DSC if you change the colour profile to AdobeRGB.

The DSC never bothered me, and the only meaningful change you can make in camera are those three letters.
So you can't make filenames in camera something useful like D300_1234, or D70s_1234 .. or whatever.

I've seen folks that use their transfer software change filenames to very meaningful types, like Sydney Opera House xxxx, and whatnot, but not interested in stuff like that.
I use folders and tagging for specific descriptive purposes.

One thing Thom Hogan harps on about constantly is camera file naming, and on that topic he's fully justified in his relentless drive to change the way cameras do this.
At least if the default naming was date/time dependent, there's be close to zero chance of replacing files over others in a multi camera user environment.

So something like 20190101000001 as a file name is less likely to be overwritten by a similarly named file which you would have had to shot at exactly 00:00:01 on 01/01/2019 as well ... highly unlikely.

From what I've read(I think .. was a long time ago), the reason our high end cameras maintain this DSC_1234 file naming has to do with some old backward compatibility reasoning for old software ... or something like that.
Something to do with DOS 8.3 file name thingie ...

[ricktas]

Have you found strange child-like drawing in your home at all AK?

Me thinks Mr Squiggle might be playing with your computer when you are not paying attention.:D:D:D

[arthurking83]

Have you found strange child-like drawing in your home at all AK?

.....

.. actually! ..

:D

[arthurking83]

An update!

It's always prudent check the operational competence of the link between the external hardware and the user input device .. in this case the mouse.

So!! .. it turns out nothing wrong with hardware, no 'loss' of data(images) ... it was the dreaded PEBCAK tech support issue.

Standard procedure for me(as I know many others do too) is to create a new folder, date-name-etc. So, on this B drive(SSD) that I use for all new sessions for the year, I had made, via Nikon Transfer .. a new folder.
Lets say I called it B:\ .. \D800E\20190106_Ararat-Mt Cole.
I use dates as the priority naming in the naming as this way they're structured in chronological order.

Seems simple enough, except that I'd been a bit hasty, and after clicking buttons, I wasn't paying attention, and Transfer2 actually sent all the images to a default location when things don't go right with hardware(or in this case .. PEBCAK)

it's default location is C:\Users\<name>\Pictures\Nikon Transfer 2. I set Transfer 2 to automagically open ViewNX2 after transfer, so I'm assuming I'm working on images stored on the B drive.
So instead of this SSD I wanted them sent too ... guess where the lost data was stored ;)

ie. I got a case of PEBCAK ... Problem Exists Between Chair And Keyboard :o

[tandeejay]

That dreaded PEBCAK error designation ID-10-T ;)

Good to hear you found your photos

Sent from my LG-M700 using Tapatalk