Management & Storage of File Size

[mongo]

Mongo would be very grateful for some technical advice about file sizes generally. First, a quick explanation of what Mongo is doing and what he wants to know/achieve.


Using a D800E in RAW full frame. This gives a very large file (unfortunately). Mongo still likes to use ViewNX2 for some very quick RAW edits - then- converts to Tiff - then - imports into Photoshop to complete the edits and saves that image as the master copy in Tiff format (16 bit) with the longest side being 3000 pixels. Notwithstanding some heavy cropping during photoshop editing, this can still finish up being a 45MB file. (Yes, Mongo knows ViewNX2 is a very slow method but it does what he wants it to do; particularly the ability to do small RAW edits and convert). Sometimes use Adobe Camera RAW.


Mongo did a quick experiment on one of the finished files by converting it from 16 bit to 8 bit and the file sized dropped from 46mb to 23mb with no noticeable deterioration in the image. Is this a viable thing or does it adversely affect the quality and ability to manipulate and reproduce the master copy later with the same quality ??

In short, is there any way that the finished file (that Mongo described in the second paragraph above) can be saved/stored as a full quality Tiff file (still 3000 pixels on the long side) but much smaller ??


All advice gratefully accepted

[ameerat42]

How about a TIFF with LZW compression? These "attain" about half their full size betimes.
Unless I've had a decade of being GROWN er, wrong, LZW is still lossless.

Hava lookat the 2nd para in this article (http://havecamerawilltravel.com/photographer/tiff-image-compression-lzw-zip). It seems to
have heard your plea.

Oh, and BTW, in your post you said "...with no noticeable deterioration..." This is an important factor in processing. If you could practise such
an effect on a continuous gamut of 16-bit color/tone I reckon you'd notice some degradation.

Am.

[ricktas]

TIFF files are notoriously large and as Am says using compression during saving can help, does ViewNX2 offer this option when saving a TIFF file?

The other thing to consider is once you have finished working in Photoshop and saved your final version, what use is the TIFF from ViewNX2? You have the original RAW file from the D800e, and you can save a high quality, full size version from PS, either as TIFF or PSD (or other format), then create your small JPG version for net use. Does the originally created TIFF serve any purpose to you in future? You can always go back to the RAW file and create a new edit in need.

If you do need the TIFF file, do you keeps layers, it can end up being a massive file. If you do not need the separate layers, flatten the image before saving as TIFF to reduce the size of the TIFF, and then save using compression.

I work using Lightroom, do some whole of image edits in LR, move the file across to PS (without saving) other than using the LR sidecar file that stores the edit information, do all my final edits, save as a compressed TIFF (though recently I have been saving as PSD to allow me to keep layers and filesize lower), resize for web and save that one. So even though I use LR and PS, the only saving I do is in PS.

Having said all that, I am glad that storage GB costs are dropping cause it is easy to fill up external HDD's etc quickly with files from a D800.

[mongo]

thanks AM and Rick

short answers are :- Mongo saved the full edited image in Tiff and discards completely the original NEF file. The Tiff is Not stored in layers and yes it is often saved using LZW compression (which is not offered in View NX2 but is in Photoshop). Mongo creates any small jpegs for net use from the master stored Tiff. So, it seems, that by taking all you have said above, Mongo is probably doing it as economically as possible storage wise.

The only other thing he can do is discover if 8 bit is as good as 16 bit for printing and use as 3000 pixel long master copy. If it is, Mongo can halve the file sizes immediately and they become quicker to work with in future too.

It would be great if you could reduce the size of the NEF before processing it. So far, all software Mongo has seen only allows you to convert the NEF to something else but not reduce it. The only other way of getting a smaller NEF is to use crop mode in camera. While this may be OK most times for distant bird or surfing images, it is no good at all when you are shooting very wide angle and want to keep it so.

[arthurking83]

thanks AM and Rick

short answers are :- Mongo saved the full edited image in Tiff and discards completely the original NEF file. The Tiff is Not stored in layers and yes it is often saved using LZW compression (which is not offered in View NX2 but is in Photoshop). ....


ViewNX does offer LZW compression as an option.

It depends on the way you 'save' the tiff file.

1/. using the simple convert to option (up in the toolbar look for the Convert to icon) .. when the window opens up, immediately below the file type option box is a tick box to use LZW compression as well, of course only available if the TIFF option is chosen(this is true for both 8bit and 16bit tiffs)

2/. if you choose the slightly quicker Open with dialogue, then VNX2 assumes you want a higher quality file to work with in the other application(which makes sense). I guess the assumption continues that once you finish editing this tiff file in the other application, you may choose to use compression for smaller file sizes from there.

My PC specs are modest by most standards(although not pathetically low!) .. and I've never really found all that much performance differences whether the file is compressed or not(up to a certain file size for tiffs).

A full sized NEF(ie. not cropped, but also not edited .. yet) using ViewNX2 to convert to tiff could yeild results like so:
(remember unedited NEF file, starting with a base 70-ish Mb)

8bit tiff with LZW saved at ... 26Mb ... 3000pixel versions for comparison ... 6Mb
8bit tiff with no LZW ............ 106Mb ... 3000pixel versions for comparison ... 17Mb
16bit tiff with LZW ............. 168Mb ... 3000pixel versions for comparison ... 28Mb
16bit tiff with no LZW .......... 212Mb ... 3000pixel versions for comparison ... 35Mb

remember tho that those files are on an unedited NEF file(at 72Mb).
Adding edits generally(ie. almost always!!) adds some extra data into the file, even if this is only more data into the exif/metadata area.
But as you add image edits, WB, colour, contrast .. etc. this in itself usually adds more data into the actual image(pixel) data area of the file.

So, I may start with a very dull looking (70-ish Mb NEF file) .. but once WB is adjusted to suit the conditions, this file suddenly inflates to about 85-ish Mb .. add a picture control contrast curve(say Vivid), and it's easy to get 95-100Mb NEF files.
While this may not be important.. the resulting tiff files may be!

A resulting tiff file from such a raw file can inflate to over 250Mb easily! But... it's not so simple to understand how/why .. or more importantly .. what the hell!
Like I said before, ViewNX2 does have a LZW compression option, but you may need to be careful with it(because of the above weird .. what the hell situation)

So as before, with this 95Mb edited file now(remember it's only added 20Mb of raw data)

8bit tiff with LZW saves too .. 45Mb .. 3000pixel version .... 8Mb
8bit tiff no LZW saves too .... 106Mb .. 3000pixel version .. 18Mb
16bit tiff no LZW saved ....... 251Mb! ... 3000px version ... 43Mb **
16bit no LZW saved to ....... 212Mb ..... 3000px version ... 35Mb

Note the anomaly with the edited and very large LZW compressed 16bit version at 251Mb compared to the uncompressed 212Mb version.
This setting was double checked and obviously again with the 3000pixel version, the anomaly arises.
it's not just this image, but I've seen it with others too. Other image browsers(mainly FastStone) also confirm the use of compression on each file, or not).
So I haven't mistaken the compressed file for the uncompressed file .. it's just that compressing the tiff file actually increases it's size! :confused:


AK83 generally is of the belief that Mongo understands what he is doing, but AK83 is still uncertain as to why Mongo would delete the more important raw file(obviously space considerations) and keep the lower quality tiff file for archival purposes.
For sure you would fit 2 or 3 times as many files onto any given hard drive, but one would assume that the retention of the better quality files are of the utmost priority.

Disregarding all the image editing and printing ability(of which the 16bit file is definitely the more flexible to add further spice too) .. the key importance in archiving the raw file(top my mind), is simply one of ownership.
You can create, manipulate and forge any file into a tiff or jpg file to suit any purpose, but the one thing that is surely impossible to do is to recreate a camera generated raw file.
I guess this is true of all manufacturers, but Nikon definitely uses some form of encryption in their proprietary raw file system. If it's not encryption proper, then it is some unknown coding system.
But once you manipulate the camera generated raw file, you cannot under any conditions get it back onto the camera ... only a unedited raw file can be placed back onto the camera to be viewed again(on the camera).

This in a roundabout sort of way can be proof that you own an image(you have the raw file, you own the image).
May not be an important consideration now, but who knows what the future may bring. imagine a possibility that someone accuses Mongo of theft of an image, which may just happen to be a coincidental occurrence that you shot a similar image .. it could be easily argued that the tiff file in your archive could have easily been a derivative of any one elses work.
If you have the NEF file as well(as proof that the derivative work is from this raw file), then no one could argue that the raw file is stolen .. (obviously)unless you stole their raw file.

Same situation if the issue was reversed, and someone stole your image .. you would ask for any perpetrator to supply a raw file(camera raw file) of the image.

Note that you can generate pseudo raw files in the native camera format, but these are not camera raw files .. hence the term pseudo raw files.


Going by your numbers .. I'd say it's not worth the 1/2 space difference in keeping the tiff file(if 45Mb is a standard average final tiff file). Your original NEF woudl be in the mid 70's Mb range(possibly 80Mb).

From that sort of file size difference .. it'd say keep the raw files, delete all the derivatives once you have finished with them.
That is, don't keep the NEF file AND the 45Mb tiff file. You can easily create the tiff file once again from the raw file. What you can't do is recreate a (proper) NEF file from the archived tiff file ever again.
Keep the NEF file .. and get a bigger hard drive.

if you insist on keeping the tiff file only, some things to note:
tiff is not like jpg, in that compression is non destructive. LZW compression is like zip file compression. jpg compression is lossy, so as you do more, it deteriorates the file. Not so with LZW.
What LZW is, is like a zip file that is seemelessly and automatically compressed/decompressd by the software you choose to open the file with.
you could gain some space by zipping up (or 7z, or any other compression software) your tiff file archive .. manually.
As an example(on tiff files), using 7zip(in 7z format), for the two large tiff files, I can compress them from their respective 251 and 212mb sizes, down to 221 and 173mb respectively .. approx 30-40Mb(10-15%) saving per file.
Obviously not a massive saving, but if your archive is in the terabytes, that's 100-150 gig saved(if all tiff) .. which of course adds thousands more images.

[mongo]

Arthur thanks so much for going to the trouble of a very comprehensive answer.


First, Mongo has the same “what the hell” experience you had when playing around with edited and unedited View NX files.


Unedited NEF - about 35mb
Edited NEF in view NX - now about 8omb !
convert that edited NEF to 16 bit Tiff in View NX - you get about 210mb !!!


Secondly, Mongo discards the original NEF because once he has edited it, he will only ever use the edited final master Tiff copy. Its that simple ! Have to disagree with you when you say that it is easy to recreate the Tiff file from the NEF. Sometimes, hours of post processing go into creating the Tiff master copy from the NEF and Mongo would NOT like to have to do those edits every time he wants them from the NEF - that is not an option. In this regard, someone has suggested that it is possible to keep the NEF file with the edits in a side car. If that is possible, then, it would seem the best way provided it is not bigger than a completed master Tiff file. Must look into this possibility .


Thirdly, must say that Mongo had not thought of the proof aspects of keeping the original NEF as you have explained. This is worth some serious thought and Mongo thanks for this important aspect.


Lastly, Mongo thanks you for suggesting a way of zipping Tiffs for storage. There seems to be a non destructive storage saving that way too.




regards




Mongo

[ameerat42]

I must say that TWO points jumped out at Am when reading this.
1. That Mongo gets rid of ORIGINAL NEFs - but Am's ideas have been covered by AK, and so no further comment that would be useful.
2. That simple edits INFLATE NEFs so much!!! Am checked with his humble 15MPx Σ files and found they had very little if any file size change - always ~40-45 MB.

Am can retrieve the original original raw file info easily, seemingly by the programming ignoring the edits done. Am thinks (??) these are stored as xml data somewhere on/in the file.

Hmm! Interesting.
Am(<- him).

[NikonNellie]

Mongo - can't believe you don't keep your original NEF's. What if you decide that you want to rework your images using a different style of editing? I am constantly going back over my original NEF's and trying new processing techniques that I learn along my photographic journey. I have recently been reworking some of my images from my D80 camera - I am using better techniques than the ones that were around when I had my D80, i.e. better ways of sharpening, noise reduction, bringing out colour, vignetting, etc.

My workflow is similar to Rick's - I initially use LR to make adjustments to the raw file (occasionally I will use a preset for a certain look), then I open up that file in PS as a TIFF. Once I have finished processing it, I save this file at 16 bits for archiving and then I resize and sharpen the image for web use and save this as a High Resolution jpeg. I currently have three storage devices equivalent to 1.75tbs and still have some space on them. :D

[arthurking83]

..........
First, Mongo has the same “what the hell” experience you had when playing around with edited and unedited View NX files.


Unedited NEF - about 35mb
Edited NEF in view NX - now about 8omb !
convert that edited NEF to 16 bit Tiff in View NX - you get about 210mb !!!
.........

Hopefully I get this right, as it was many years ago that I noted similar things. But they aren't necessarily what the hell moments.
More so, some intricate processes that Nikon have implemented for our convenience.

To be sure, my what the hell moment is simply that with compression(LZW) the tiff file is larger than the uncompressed tiff file(no LZW) applied.
To me this simply makes no sense at all .. ie. what the hell.

In your situation, the answer is more likely to be simple.

Unedited file straight off the camera is 35mb .. most likely reason is you have set compression on the raw files in camera.
This is good! I would maintain this workflow if space is an important aspect. It keeps the original raw file to a low space requirement.

ViewNX2 can obviously read these compressed raw files, but then (again I'm assuming that Nikon assumes) you want the best quality file to work with.
it(nor CaptureNX2) can compress the raw files again tho.
Once you start working with them, the compressed files are inflated to their optimal data and stay this way.

Converted tiff from raw file .. 200Mb is about normal. Remember that any editing adds data to the raw file.

The raw file is a super compressed tiff file of some type. Proprietary Nikon file tho remember.
Most raw files from all manufacturers seem to be tiff files of some type, with manufacturer encoding/tweaking/compressing done to them.

Raw is the most efficient file type to archive .. it has optimal data AND low space requirements.
obviously other files types are smaller again, but at the expense of data.

Check the camera for compression and bit levels set.

if you want the best file type possible but also the lowest file size too, try setting raw file type to 12 bit with compression added.

This of course depends on your shooting style/method and possible future needs.
The general consensus is that the difference between 12 and 14 bit raw capture is that the last 2 bits(which add significantly to the raw file size) are all lost in the highlight region.
Again going by general consensus, we can't see those last 2 bits of lost info so there's nothing really gained for most of us in using 12 bit mode.
But the computer(or more accurately the digital realm) can see those lost 2 bits, and where it supposedly comes into the equation is if you push the dynamic range harder in PP.
That is, if you shoot for maximum dynamic range retention in a single file, deliberately blow the highlight regions to capture more shadow data, and then recover in PP.

I do this almost all the time if the situation calls for it, even taking into account that I stuff around with filters to capture maximum dynamic range.

if you don't work in this manner, then 12 bit and compressed mode may yield low enough raw data to make archival of those raw files a serious proposition.

ps. I just tried to do a compression test on some raw files using 7Zip.
i didnt' think they had much leeway to archive with compression, but the results were surprising.

1.98G of data in 25files(file sizes varied tho .. from 71Mb to 101Mb) all NEFs.
7Zip compressed them from 1.98G down to a total of 1.2G.
My math may be out, but I think that's a pretty good % rate at about 30% reduction.
Took a long time tho, even with all four cores doing the work ... about 15mins or so .. can't remember, as it was smoko time too during this test :p .. cough, cough wheeeez!

Quick raw capture test with varying bits and compression levels.
Scene is a simple colour type. Remember more colour = more data capture! .. and dynamic range was medium.

14bit (no compression) raw in VNX2 = 74Mb .. edited = 93Mb
14bit (lossless comp) ... raw in VNX2 = 40Mb .. edited = 58
14bit (lossy comp) ...... raw in VNX2 = 37Mb .. edited = 51
12bit (no compression) raw in VNX2 = 50Mb .. edited = 74
12bit (lossless comp) .. raw in VNX2 = 34Mb .. edited = 53
12bit (lossy comp) ........ raw in VNX2 = 31Mb .. edited = 50

al edits were the same type, applied in a batch. WB and setting Picture Control to Portrait(maximise dynamic range again). So no file was favoured over any other to bias the file size inflation.

But as we can see, the compression of the original files has been 'undone' to a degree.

From what you can see here, there is a good case in using lossy compressed 12 bit mode in raw, as long as you don't push processing too hard(or more accurately don't expect to push PP too hard).

my test involves a very simple scene. flat colour and not a lot of it.
But if the scene is complex in colour and tonal range .. imagine a very nicely coloured landscape scene .. or intricately coloured bird .. the amount of compression must surely vary in the camera, so the amount of decompression(inflation) will be concomitantly variable .. and so in a more colourful higher data capture scene, the compressed file will surely increase by a larger margin that I have posted here.

if 30Mb isn't too large a file size for 'ya, on a per file basis .. I'd say shoot in that manner.
Keep an original version of the file somewhere for archiving. it's small enough at 30-ish Mb to not be intrusive(me thinks).
But remember, once you open that file and edit, the recompression effect of VNX2 is permanent, and it will inflate to 50 or more Mb and you can't go back .. even if you reset the NEF file back to it's original in camera conditions.
So work on a copy of that file, save it to the tiff format of your choice for further editing and then delete the worked on NEF to save space .. you already have the 30mb version, and you can easily create another 50Mb version later if really needed. The current 50Mb version that you worked on is subsequently rendered surplus to requirements.


The other point to be mindful of when using VNX2 when editing .. all edit steps are stored in the actual raw file being worked on.
This is my preferred method of working with raw files for many reasons, all of which are not related to the issue raised in this thread tho.
But the point is that there is no external sidecar file being stored anywhere.
Any added IPTC (ie. metadata) is also stored internally to the actual raw file.

Other software(s) use the now more common external sidecar file system, but to create a image preview of the edited image jpg files need to be created to view thumbnail versions of the edited raw file.
With the old Nikon system (VNX2/CNX2) of internally storing the saved edits, the preview image is also stored internally to the raw file.
What this means is that if you edit the raw file in VNX2, and want to sort through you images with any other software .. eg. a file browser of some repute, then you will see the actual edited version of the raw file.
(because the preview file is stored within the NEF)

If on the other hand you choose a software type that uses the external sidecar file creation method, you lose the ability to see the actual edited image that the raw file is supposed to be, but you have no gain in the edited raw file itself(because all added data is kept separated from the original file).

Note that ViewNX-i now uses this external sidecar file setup .. but is totally useless for edits as I assume you do with VNX2.
VNX-i has no editing ability, other than to pass the file onto CaptureNX-D .... which is the true successor to ViewNX2(not Capture NX2 as Nikon tried to explain!).

If CNX-D actually worked .. as in didn't crash 5 times out of every 4 :p .. I'd recommend you migrate to this software to continue with the same type of VNX2 style basic editing you are doing now.
As an editor in that sense, it's much better .. more powerful, with better tools.
But the problem is speed and stability .. of which it scores -10's on both counts.

ViewNX-i lasted all of about 10 mins on my PC, and had to be removed so that I can maintain use of VNX2.
I keep CNX-D only because I have a belief that one day hell will freeze over and Nikon will finally give us some useful software that does what we need it too do(with the appropriate tools)! :p

[mongo]

Mongo - can't believe you don't keep your original NEF's. What if you decide that you want to rework your images using a different style of editing? I am constantly going back over my original NEF's and trying new processing techniques that I learn along my photographic journey. I have recently been reworking some of my images from my D80 camera - I am using better techniques than the ones that were around when I had my D80, i.e. better ways of sharpening, noise reduction, bringing out colour, vignetting, etc.

My workflow is similar to Rick's - I initially use LR to make adjustments to the raw file (occasionally I will use a preset for a certain look), then I open up that file in PS as a TIFF. Once I have finished processing it, I save this file at 16 bits for archiving and then I resize and sharpen the image for web use and save this as a High Resolution jpeg. I currently have three storage devices equivalent to 1.75tbs and still have some space on them. :D


thanks for the advice Nellie.

As Mongo said, once the NEF has been edited fully to the end point Mongo wants to get to, it has no other uses (apart form what has been said about proof of ownership of the image). The whole point of doing PP on a RAW image is to get it to its best and then store it as a lossless Tiff which can be used as the completed master copy. Mongo does not rework his images. He gets them as good as he can the first time and then saves those changes as the new completed image and discards the original "draft RAW" image. Mongo does not want to go back time and again to do PP on the original RAW image after he has already done that and stored it as a Tiff. Not worried about external hard drives - have plenty of those but it is a matter of convenience and file size is easier and quicker to work with if it is smaller. That is what has spurred this thread.

- - - Updated - - -

Arthur - thanks again.

Mongo has only just started using 12 bit lossless compressed setting in camera in the last few days. Lance was kind enough to put me onto this a few days ago. That will help.

The NEF with sidecar sounds a good idea. However, as Mongo understands it, the sidecar only carries the edits made in RAW using , say, View NX2 and the like. The problem is that Mongo goes on from that point to do 90% more of his editing in Photoshop. That is why he has to store those substantial edits and the way to do that is to store them as a new Tiff file. Those photoshop edits cannot (as Mongo understands it) be saved in the NEF sidecar. In that case , all the real edit work would be lost if not saved asa Tiff and Mongo would have to start the editing process all over again each time with the original NEF file( and whatever small edits had been saved in its sidecar). Mongo can spend hours on an image at times and it is inconceivable that he would just was all that time and work by not saving it in the only way possible - eg a Tiff master copy of the completed image. The NEF file simply cannot even come close to saving that PP work.

30 mg files are not a problem. The whole exercise is to get the files as small as possible and still retain the quality and edits. If 30 mg would do that, Mongo would be more than happy but there is no answer from Mongo can see to the problem he has posed.

appreciate everyone's advice. It at least confirms that there is no answer to Mongo's problem and that in itself is worth knowing - at least until some new software comes along that lets you save photoshop edits/all edits as a sidecar to the NEF file