"RAW" processing and various programs

[ameerat42]

(I hope!)

I recently installed a swag of programs to see how they handle the output from a friend's (one who is just starting out with a DSLR and a vintage one at that) Nikon D70.

At first I had ViewNX, from Nikon, as I understand.
Now I have installed RawTherapee and the latest, FastStone Image Viewer.

They all open up the output from the D70 that are listed as *.NEF files.

The Q is: what are these programs actually looking at?

And further, Is it the NEF (raw) file itself, or is it something called the embedded jpeg?

Note that the settings on the camera are to output raw only.

My suspicions are that the ViewNX program actually opens the full raw file, but I am not confident as to anything else, especially when the word "view" is attached.

Any insights would be appreciated.

Am.

[I @ M]

Am, the NEF file extension only applies to Nikons proprietary raw files.
View NX ( you should be using the "2" version ) is simply reading the NEF file and rendering it as a viewable jpeg.
View NX2 is capable of and does unpack the entire NEF file and allow edits and resaving as either a further NEF, a tiff or jpeg.

The easiest way to see what is happening is to grab a windows based machine without the correct NEF codec installed and drop an NEF file into it and try to view the image through windows explorer. You will see no image from the NEF file but all jpeg and tiff images will show.

[ameerat42]

Ta, Andrew. Yes, it is the ViewNX 2 version.
I thought that was the case with this program - as you describe - unpacking the raw file.
I have heard pros and cons about what the others might do, like RawTherapee and Fast..., ie, whether they only open an "embedded jpeg" (whatever that means) and let you "work" on that.
Certainly, I cannot save an opened NEF as a 16-bit tiff using FastStone, but it does allow saves as a DNG (16-bit, I suppose).
Am.

[arthurking83]

ViewNX displays the jpg file on screen, but offers the ability to work/edit the raw file, as it's displayed on the camera.

FSVFiewer has a couple of options for viewing the NEF file.
It can be set to view the embedded jpg file, so it will be rendered on screen as it is on the camera, or it can be set to review the actual raw file data, which it then displays on screen with it's own representation of the raw file.
This will almost certainly appear differently to how the camera and VNX will display the file!!
So bear in mind, if you edit the NEF file with Nikon's software, the software alters the embedded jpg file to suit(the processing).
If FSV is set to display the embedded jpg file, then that NEF file is rendered as edited by VNX.
BUT!! if you set FSV to preview the raw file, the edits on the raw file will not(should not!!) display as they were set in VNX. So what you see in VNX will not look the same as it will in FSV.

To set this option in FSViewer go to Settings(toolbar) Settings(scroll down the list) and then look for the Raw tab in the settings menu. It will say that displaying the full raw file will slow down the program for each file that is viewed .. and this it does!! .. takes a few seconds to render each NEF file as they are highlighted. Whereas if set to display embedded jpg files, the program is lightning quick(one of the fastest I've used for previewing my images).
Note too tho(on my PC) ViewNX2 has now become the fastest NEF viewing software I use to flick between my raw files.
It has it's foibles, and I still use FSV for the odd purpose(eg. comparing images side by side, etc) .. so FSViewer is still one of my faves.
So much so that I've installed it on our work PCs for those times when we need to capture work on and email images to and fro. .. and I have it on my USB thumb drive too! :th3:

Note too tho .. no matter how FSViewer is set in the settings(with respect to raw file rendering), when you convert raw files in FSV, it wil always do it's own raw to <image file> conversion .. so that the final image will not look like the preview image(if set to preview embedded jpg)

I've never used RAW Therapee, so I can't comment, but so far that I've seen, all third party software seem to work the same way .. they very rarely render a raw file as the native software, or cameras do.
(actually, I think I have at one point but wasn't overly impressed, compared to what I already have/had).

That's one of my major prerequisites for image software .. to render the raw file as I captured it on the camera.(fewer processing steps)

[ameerat42]

Thanks, AK. I will now set to and digest this. I must have missed that option for raw in FS, but I will go back and start there.
Am.

[arthurking83]

OK, scratch that!!

I just did, and it did too! :th3:

I originally said that FSViewer doesn't save the NEF file, using the embedded image file data .. jpg rendering and all and maintain the same renderign as the Nikon softwear rendered raw file.

But I just tried it, and it does!!

..... note I haven't done this for a good many years after noting this 'discrepancy' the first time I tried it on my D70s raw files.

Good to see that it now converts the raw files as edited and saved in Nikon software.
I have no idea when they changed this, but the earlier incarnations of FSViewer I'd used, didn't.

Also another point to note about Nikon's software! They're not the smartest programs out there for viewing all manner of image types, so the reality is that they're specialist software for maximising Nikon raw files.
I never use Nikon software, where I can avoid it, for viewing non Nikon made jpg files(eg. my daughters Samsung P&S jpgs!!).
One of the issues with my Nikon software usage is that it doesn't always render TIFF files, or any type. They have to be set in a particular manner, and FSViewer is one of the best for doing that.

eg. If I use PTGui to render a pano, and save to tiff(as I always do), I don't set the tiff file with any compression, no alpha channel(whatever that is). Any deviation from a raw as hell tiff file, and Nikon software throws a fit .. won't render the file.
Even tho it has the ability to set LZW compression itself, if this is done with most other software, again, Nikon software will not view the tiff file.
But as long as this compression is done using FSViewer, the files render for me ... :confused:
(another reason I still use FSV).

Even tho FSViewer doesn't allow you to save in 16bit TIFF, use the 24bit setting.
It must be a 16bit setting with some added fluff of whatever kind, as Nikon only recognise 16bit tiff files(due to it's limited ability) and these 24bit saved files render perfectly in Nikon software.

I have no idea what the extra 8bit in FSViewer's settings are nor what they do or allow, but Nikon's software will still render them perfectly .. where it has trouble rendering tiff saved from other software.

[ricktas]

Another thing to remember is that the RAW file is just raw data. What these programs do is collect and collate that raw data and turn it into a visible representation of that data. What you see on screen is often (not always) a TIF file version of the raw data. But it really does not have an actual 'file format' till you save it.

What a raw converter does is it converts (funnily enough) the raw data into an image that can be displayed on your computer monitor.

Quote from THIS (http://en.wikipedia.org/wiki/Raw_image_format) article:

Many raw file formats, including 3FR (Hasselblad (http://en.wikipedia.org/wiki/Hasselblad)), DCR, K25, KDC (Kodak (http://en.wikipedia.org/wiki/Kodak)), IIQ (Phase One), CR2 (Canon (http://en.wikipedia.org/wiki/Canon_%28company%29)), ERF (Epson (http://en.wikipedia.org/wiki/Epson)), MEF (Mamiya (http://en.wikipedia.org/wiki/Mamiya)), MOS (Leaf (http://en.wikipedia.org/wiki/Leaf_Medium_Format_Photography)), NEF (Nikon (http://en.wikipedia.org/wiki/Nikon)), ORF (http://en.wikipedia.org/wiki/Orf_format) (Olympus (http://en.wikipedia.org/wiki/Olympus_Corporation)), PEF (Pentax (http://en.wikipedia.org/wiki/Pentax)), RW2 (Panasonic (http://en.wikipedia.org/wiki/Panasonic)) and ARW, SRF, SR2 (Sony (http://en.wikipedia.org/wiki/Sony)), are based on the TIFF file format.[2] (http://en.wikipedia.org/wiki/Raw_image_format#cite_note-1) These files may deviate from the TIFF standard in a number of ways, including the use of a non-standard file header, the inclusion of additional image tags and the encryption of some of the tagged data.

[arthurking83]

Also note (and sorry for the ranting) but it may be important to any elf confessed wannbe geek!) .. when FSViewer is set to 'display embedded jpg' .. it does in fact display the full sized embedded jpg file.

There are two embedded jpg files, one thumbnail(small size) and one full sized jpg file(large file).

For pixel peeping display, it displays the large file .. for the thumbnail view it uses the small file.

The reason for explaining this is for those times when(if) you make alterations to the colourspace of the raw file. It can create discrepancies between a thumbnail image(small file) and the actual full sized view(large file).

But you(or your friend) may never encounter this issue as this is more specific to CaptureNX and the ability to set colourspace as you edit the raw file.
ViewNX doesn't allow this colourspace adjustment on a per image basis, and is only a global setting tweak.

[ameerat42]

Ta AK and Rick.
Yes, I am finding that FastStone - at least so it seems - is able to actually open the raw file (went to the RAW setting in the F12 menu).

AK, as for the 8 and 24-bits, I checked the properties of a saved tiff and found it to be a 48-bit file, ie, 16-bits/channel. Now, the only thing I can't remember is what program I used for it.
(Now I have to go back and try it again.)

I've got to work out an ideal program/workflow for this friend as he tends to want to take pics more than process them.
Ta, Am.

[WhoDo]

I've got to work out an ideal program/workflow for this friend as he tends to want to take pics more than process them.
I think you should avoid Raw Therapee in that workflow, Am. I've used it and I've found it doesn't always produce acceptable results in the conversion process. If you need something free, other than Fastone, to perform Raw file editing in Windows then I'd recommend XDepth Raw (http://raw.xdepth.com/xdraw_news.php). It will also interface with GIMP if your user doesn't have any other image editing software. :th3:

[ameerat42]

Thanks, Waz. I'll look at it in a spare mo.
I think that since he has had to consider the reality of some PP he is getting more interested in ViewNX, in fact, but I'll have a look.
Am.

[arthurking83]

A workflow option for your friend to consider could look like this:

VNX2 for quick sorting, quick touchups(on NEF files, not jpg or tif files!!) and setting some keywording if he's smart.
The keywording option is easy to do, it's right there on the edit pane (RHS) under the editing options.
(I'll explain why keywording in Nikon's software MAY become important).

Save any quick easy simple edits and convert the image to tif. VNX is one of the fastest NEF to tiff converters available .... except for the old Bibble, which transmorphed into Corel's new After Shot Pro( have never really used ASP myself).
From there, you'd set a quick launch icon in VNX2 to open the image in GIMP and edit away to your hearts content.

GIMP also has the ability to allow raw plugins such as UFRaw, which is DCRaw in disguise and many people complement DCRaw's abilities.

But!... here's a hypothetical:

Say you get into using LightRoom(only an example, and nothing to do with software bashing or belittlement!)
LR (Adobe software in general) writes all it's edits into a side car file(XML) and is really only readable by Adobe software .. but I suspect that other software manufacturers will have some software that can convert these side car files into their own formats.
But lets say you tire of LR after 6 solid years of non stop processing.
As the raw files themselves contain no info as to the level of work that's gone into editing them over the past 6 years, and you decided that you've found another raw/image editor, you are now exposed to a limitation that this new editor needs to have the ability to convert a proprietary side car file into a format of it's own choosing.

ie. after years of use, you've locked yourself into a system of software due to the logistics of moving your digital warehouse over to a new system(hope that makes sense!!)

Nikon's edits are contained within the raw file itself, but are still separate sidecar files.. the difference is that the side car data is contained in the raw file.
So(the end result) is that as long as the software has the ability to read or display the raw file as it should be, then any edits made in Nikon's software is displayed as edited.
i.e. you aren't locked into a software system :th3: .. as long as your new software has the ability to read the Nikon edited raw files as they have been edited.

If a free program like FSViewer can be setup so that it can read the raw files as edited, then a paid for program SHOULD be able to do so as well!
One of my main gripes with LR .. it can't read any edits made by Nikon software from the internally contained sidecar data.
FSViewer doesn't have the ability to read LR converted or edited raw files, so if you choose to view an LR edited raw file, you have (that I know of) two alternative solutions for viewing those raw files directly. Both Adobe software.

People claim that the raw file should be left untouched(which is a load of garbage, to begin with!!) .. but the reality is that the raw file is left untouched.
The edited data is simply contained in a separate area in the raw file, and can always be undone at the click of a button(in Nikon's software).
How it operates is that you haven't directly affected any raw data, you've set a load of instructions for the software to render the image in a particular manner and this is written to the embedded jpg file).

Therefore the onus is on the software manufacturer to code their software so that it reads these instructions, (or not, as the owner chooses).
Once again, it comes down to not being locked into a software system, if the tide changes and better software is made available in the future.

So again(to be sure my post is understood), this is not a bash at LR, as my quick play with ASP and other software has been the same .. they're too limited in their ability to offer the owner the flexibility to do as they please.

I have hundreds of thousands of image over the years myself, I don't want any software manufacturer to dictate what they think I want and need. .. they should be pandering to my, and yours and everyone elses needs.

Understand the guts of the workings(of raw files), look ahead at how it may affect your long term prognosis and goals, and maintain the freedom to change at a moments notice.

FWIW: I see why people like LR, and I have it myself and mainly for some odd spot doodles, and for the ability to read 'alien' raw file formats to assess other options(Nikon's software is ONLY for quality manipulation of NEF files .. at anything else is quite the useless contender.
LR has very easy to use noise reduction ability compared to what I've been used too.
But as already said, I can't find any way to set LR to read NEF files as they've been set, where this is a simple three click process in a freebie like FSViewer. So it's not that it can't be done due to proprietary Nikon grumblings and groans .... it's simply a failure to offer the owner this option.

apologies for the rant, but I'm hoping it explains a bit about my choices and suggestions.

[WhoDo]

I take your points, Arthur, but aren't you the least bit nervous about Google owning Nik software now? ViewNX2 might be the last of its kind, or they may decide to take the package in a whole different direction. :confused013 If your argument holds for ViewNX2, then maybe the editing should be done in Capture NX2 as well; after all Andrew (I@M) swears by it and wouldn't use anything else.

[ameerat42]

TA most lotly, AK. I actually understood what you were saying. (Boy! And I thought I was exempt from Nikonery. It's only thanks to this friend that I find myself immersed in it.)
However, it is not a real lament. I find it interesting and it broadens horizons.

Am.

[arthurking83]

Yeah! for sure there is uncertainty over CNX2.
But me thinks that Nikon will supply an out clause at the least, so that whatever new program they develop(with or without a new partnership), it will have backward compatibility with all old Nikon software developed/edited images.

This was one of the points that I did like about Nikon's change from Nikon's old software, Nikon Capture(not CaptureNX!!).

the editing was entirely different and so the edit steps were different from Capture(which finished at v4.something), and then morphed into CaptureNX (v1).
You loaded your edited raw files(edited in Capture) in CaptureNX, and the edits were maintained faithfully.

The problem is(and usually is) that if you made edits in CaptureNX, they couldn't be displayed in Nikon Capture.
Same with the move to CaptureNX2 .. CaptureNX couldn't recognize all edits made in CNX2 .. some, but not the newly developed stuff like healing. So if you added a healing step in VNX2, a warning would pop up explaining that the edit step will not be recognised, and the image displayed in CNX but without the healing step.

I'm at ease with whatever Nikon's next incarnation of editing software becomes, it will maintain compatibility with the editing already made to existing NEF files.
(how other software reads those raw files is another question tho!).

One problem with this uncertainty over Google's ownership of Nik, is for those of us with the Nik Color Efex plugins!! :confused013
Those plugins are 'temperamental', and not entirely Nikon's fault.

if you don't have the relevant plugin to render the plugin's editing step(within CNX/CNX2, you can't do anything with those particular files. A large watermark is presented over the rendered image, and you can really only undo those plugin edits. Any other CNX based edit steps are fine tho.

I think (as far as I'm aware) Nikon has been faithful so far in offering the ability to maintain compatibility over the past 7 years that I've been using their software.

Prior to getting my D70s, I used to use PS(and before that, PSP 6 and 7) for editing tiff file scans of our film images.
So I had some limited experience with early versions of PS.
It seemed to be easy enough to use, but I almost certainly never got maximum value from it due to my lack of interest in learning all these intricate steps and processes.
Initially I went from trying Nikon Capture 4.2 and whatever else, to preferring PS for most stuff. But I hated taking a photo, and not seeing the image as I expected to see it on the PC, as I captured it on the camera(histogram and tones). Eventually as I learned how to handle manage and understood what a raw file was capable of in terms of editing, I gravitated more towards Nikon Capture.
Not long after that decision, Nikon released CNX(v1) and that was so easy peasy to use in terms of editing processes, I never looked at PS for about 2 years after that.
Then the ex left home for greener pastures and my copy of PS somehow ended up in her kitty(my fault), but with no regrets.

I think it's handy to have a basic understanding of editing in various software to begin with, learning how to do small steps first, and then playing with more options until you have a clear understanding of what's happening and how it affects the future(of your stash of images).