RAW Conversion musings

[Aussieden]

Another thread got me thinking about RAW files and their conversion.
Whilst not a technical person I think I have a reasonable understanding of the RAW file format and the differences between RAW and JPEG.

This thread is not meant to be a debate of RAW vs JPEG.

Its about trying to understand how each third party RAW converter interprets the RAW file produced by the camera.
I read somewhere that most in house software use the in camera settings selected at the time the image was shot as the starting point which then represents the preview image on your screen.
But most third party RAW conversion software, do not apply the in camera settings.

My understanding is every third party RAW converter chooses a set of default adjustments as its starting point when it first demosaics a file. Most (all) third party RAW converters will also apply other default adjustments when opening the file such as exposure, lens corrections, denoising and sharpening.
So the same RAW file opened in any number of RAW converters will produce different results depending on the software’s interpretation of the information the camera has embedded in the file. So what is really a RAW file?

I wonder how the RAW developers engineer the algorithms and come up with how their program interprets the RAW file?
Do they then look at the image on screen and make a judgement based on what they see?

This article : http://www.nomadlens.com/raw-converters-comparison
does some interesting comparisons of RAW converters and if you look at page 2. (Defaults) each one appears different.

Nevertheless it would appear then that all the results become subjective through the eyes of the viewer depending on their taste of contrast, colour, noise, sharpness etc.
Other factors also come into play such as viewing environment, quality of monitor, colour calibration etc.

So I suspect if 10 people viewed the same RAW file on the same monitor in the same environment using a number of different RAW converters at their default settings , they would have different preferences as which converter produced the most pleasing image.

Even creating your own presets or camera profiles becomes a matter of personal taste.

After trying a number of third party RAW converters, I have lately reverted to using Olympus Viewer 3 to convert my Olympus RAW files (.ORFs).
Whilst it is fairly clunky and slow with limited exposure corrections/features, to my eyes the Olympus software renders the raw files in the most pleasing way.
I rarely make any changes to settings in OV3 but have the option to do so if required. I then export them as 16 bit TIFFs to Photoshop.
Basically this is my starting point to make further adjustments/enhancements in Photoshop as I consider are necessary.

Any thoughts/comments on RAW conversion, RAW converters you prefer would be welcome.
Dennis

[ameerat42]

"Not a technical person", eh? You must be his brother:D

After some years I think I'm getting the impression that the proprietary raw converters do about the
best job in interpreting the data unique to that brand of raw file. I've only had LTD experience, using
ViewNX and Sigma Photo Pro, but from the reports of lot of others I get that idea. And especially when
there are as many raw file types as there are image sensors - I think too.

Another point from your post: when you save in 16-bit tiff for later work elsewhere, it is still not the same
as having the original raw data (from readings). If it has preserved the gamut of tones and colors, then the
other program will treat them differently than would the raw converter.

For this "reason" I now do all but the merest tone conversion in the raw converter (and geometrical things like
cropping, distorting, etc) elsewhere, usually Photoshop - which is quite useful, but falls short of being a pocket miracle.

Nothing else at this stage.
Am.

[Cage]

This thread has 'AK' written all over it.

Give him both barrels Artie. :lol:

[arthurking83]

LOL Kev :D

Because I hate change a fair bit when the change is from an ideal situation to one that subsequently isn't, there's a bit of a back story to my reply here:

Yeah, I'll be the first to admit it's one of those topics that I'm usually interested in, but probably only for the situation Nikon have set up now more, than anything else.
Before, Nikon had their raw converter software(s) and it all worked(at least for me and a couple of others I know of).
But then Nikon changed it, and their new software path is not so much a path as it is a non existent descent into oblivion(unless they have more plans for their current software .. unlikely but hopeful!).

So now, if I ever acquire a new camera(Nikon) I'll have to start using new software to convert raws into images.

For Dennis: you're using Olympus's software(OV3) you say .. I'm curious have they set it up so that the images on the PC look exactly as they did in the camera?
So what that's asking is, the OV3 software will almost certainly use the same picture style/tone curves as the camera does, so the rendering looks the same, but I'd expect that the histogram(ie. exposure) will also look the same too.
Nikon do this with their software.
When you view a Nikon raw file in Nikon software, the tone curve and exposure levels are identical .. so the nett effect is that what you shoot in camera is what you see on the PC at the editing stage.
With ALL thirdparty raw capable software this is not the case. And I've tried many .. a plethora of many :P
None of the thirdparty raw converters allowed the option to set the edit info into the raw file itself.
There are a ton of other reasons why I don't like most of them, and now Nikon have altered their newer software to the same underlying workflow method as the thirdparty software in that they use external xml file type for holding edit information. Prior to this all the edits were held within the raw file itself so saving and cataloguing your raw files was easier, and to maintain backups of.
What this also did tho was that as you updated the edit, the embedded jpg preview file maintained previewed the point to what edit status you were at with the file.
Now tho, with the use of these external edit file place holders, you now have to use the software that created them to view 'at what point' you have edited the raw file.
With Nikon's older method, of internally placing the editing steps and updating the jpg preview file in almost any software you could see at what stage your raw file was at in terms of editing.

I've been trying to understand what substances Adobe's ACR engineers are on when it comes to exposure levels in Nikon raw files, but have given up.
Exposure is not only way off to what the camera indicates, but miles from all other raw converters too.
In general I've found that they'll be about -1Ev off what the camera and Nikon software render for most files, but half the issue is that this is not consistently the same.
Most other raw converters were only just slightly off what the camera(and Nikon software) would render.

I remember a long convoluted argument over this topic on another forum(no longer running) with an extremely opinionated(more so than myself! :p) member that was adamant that the issue was Nikon's, and not Adobe's.
Her argument was that the reason for the discrepancy was due to Nikon using a lowly 8bit jpg preview image histogram to preview exposure on the camera for any image .. including raw files.
That this 8bit only image can't truly show the accuracy of the actual raw file exposure.
While in part her comments were true, in that the histogram shown is that of the jpg preview file .. the reality was that the raw file displayed in Nikon's software displayed almost the exact same exposure and tone levels leading to the same histogram curve.

So that is, if I open the raw file in Nikon's software and compare that rendering to the one of the 'supposed' inferior jpg file shown on the camera's review screen, using Nikon's software they are, for all intents and purposes, identical. There is the most insignificant differences, that are probably measured at less than 1%, and usually on the green channel.

Basically it all boils down to what the software engineers can do with their software. Also do they care how closely their software can imitate the manufacturer's rendering .. etc.

My take on this topic is that photography has changed to what it was in old film days. In the film days, your camera was a more simple tool in that it never developed the image, it only exposed it and film was standardised in terms of exposure sensitivity.
With digital, that changed and now your camera has the power to develop the image too(which they all do).
They all have built in tone curves and exposure settings. While the ISO standard is a standard and all manufacturers adhere to similar principles of this standard, there is no hard set exposure standard they they all use. They all have a certain amount of leeway in how they use this standard .. AND it only applies to jpg images, not raw files.
I guess this is why they all use jpg files for preview and why that is the underlying reason that this histogram is used for display purposes only.
That is, that part of the workflow makes sense.
So the rendering of the raw file should really reflect that particular standardised environment(ie. not the other way around)
It makes sense to base the raw file rendering using that principle, because raw files can be so different. There are no standards set on how raw files must be handled either in camera or in software.

If I do ever acquire a new camera(which will only work with Nikon's newer software), I'll end up working with new raw converter software, and my two gripes with Lr are that you must go through their annoying cataloging/importing silliness first and that it'll take more work to get the file to where it should be in terms of exposure/tones.
I liked Capture1 in terms of rendering and the point that it didn't force you to go through the import/catalogue step, but the tools were a little more convoluted to use overall.
Still made good images tho, just harder to make the jump from old(ideal) to new(not so ideal) software.

As for how the engineers decide on how to interpret the raw files .. only the gods of 0's and 1's can understand this, and us mere mortal users have to agree to their decisions .. and then play catch up with more work.
I'm not a fan of tiff files for working on .. way too large and hoggy on resources and the raw file is as good as it gets.
I've also tested that there is at least few Ev(overall) of extra capacity for editing in a raw file compared to a tiff file too both in highlights and shadows if required.

Olympus tiff files may not be quite as large as say D800 tiff files(200+Meg each) .. and I can't imagine the size of Sony's 42Mp tiff files once they have true 14bit in camera capability .. and just imagine Canon's new 50Mp camera's tiff files!

I've only ever used tiff for printing high quality large prints, and for those rare times when I made a panorama image and wanted to keep the end result.

.. anyhow, good topic .. and sorry for the inflated reply :p

[Aussieden]

LOL Kev :D

For Dennis: you're using Olympus's software(OV3) you say .. I'm curious have they set it up so that the images on the PC look exactly as they did in the camera?
So what that's asking is, the OV3 software will almost certainly use the same picture style/tone curves as the camera does, so the rendering looks the same, but I'd expect that the histogram(ie. exposure) will also look the same too.

So that is, if I open the raw file in Nikon's software and compare that rendering to the one of the 'supposed' inferior jpg file shown on the camera's review screen, using Nikon's software they are, for all intents and purposes, identical. There is the most insignificant differences, that are probably measured at less than 1%, and usually on the green channel.

Basically it all boils down to what the software engineers can do with their software. Also do they care how closely their software can imitate the manufacturer's rendering .. etc.

As for how the engineers decide on how to interpret the raw files .. only the gods of 0's and 1's can understand this, and us mere mortal users have to agree to their decisions .. and then play catch up with more work.

.. anyhow, good topic .. and sorry for the inflated reply :p

Thanks Arthur for the detailed reply. I am still trying to digest it but my head hurts!

But I think we basically are in agreement on most things? :nod:

What I am trying to get my head around is what you actually see on your monitor when viewing a RAW file using any RAW converter.

My understanding is all RAW files have some post processing applied or you would not be able to view them. Therefore there's no such thing as an unprocessed RAW file. Is that correct?

With regard to Olympus RAW files and Olympus Viewer 3 my understanding is that it uses the in camera settings selected at the time the image was shot as the starting point which then represents the preview image on your screen. So I presume I am looking at the RAW file with the same settings as OOC JPEG but which has not been processed by the camera's JPEG engine?
Yet if you view the two different file formats side by side (at full size) there is no discernible difference and the histograms look the same. Then if I just save both files (with any editing done) to 8 bit JPEGs then they both look the same. So what am I really seeing?
Even when I turn off the noise filter, dial back the contrast, saturation and sharpening to their lowest settings (-2) there still appears to be some underlying processing being done.

The only way I can achieve a flatter file is via Lightroom reducing the noise reduction and sharpening sliders to 0.
It’s a bit difficult to demonstrate without actual images posted due to the limitation of size on this site.

Even if I posted full size images I dare say everyone would have differing opinions on which images look better. ;)
This has tested my brain matter too much, I think I need to have a lie down!

No need to reply!
Dennis

[ameerat42]

What I am trying to get my head around is what you actually see on your monitor when viewing a RAW file using any RAW converter.

--And without asking an imaging engineer you'd be hard pressed to get an answer that would make sense, so the following is just what I "reckon".

On your monitor you are seeing a representation of what the viewing program (raw converter/viewer) can interpret and what the monitor's gamut and color capabilities can show.


My understanding is all RAW files have some post processing applied or you would not be able to view them. Therefore there's no such thing as an unprocessed RAW file. Is that correct?

(I said, "an engineer".) Yes, some processing. Enough to display according to my first reply. And "No". Forget the term "post-processing" for the straight display of a raw image.
Reserve that for the subsequent, more permanent changes you save in an output file, such as jpeg, tiff. MAINLY because the raw file data remains unchanged and only some interpretative
information has been added. You later change the raw file data again - from scratch - to save another as a different output file. So, there's no such thing as a PROCESSED raw file.


With regard to Olympus RAW files and Olympus Viewer 3 my understanding is that it uses the in camera settings selected at the time the image was shot as the starting point which then represents the preview image on your screen. So I presume I am looking at the RAW file with the same settings as OOC JPEG but which has not been processed by the camera's JPEG engine?

Pretty much so:th3:


Yet if you view the two different file formats side by side (at full size) there is no discernible difference and the histograms look the same. Then if I just save both files (with any editing done) to 8 bit JPEGs then they both look the same. So what am I really seeing?
Pretty well "answered" already.


...The only way I can achieve a flatter file is via Lightroom reducing the noise reduction and sharpening sliders to 0.
It’s a bit difficult to demonstrate without actual images posted due to the limitation of size on this site.

(I'm not sure what you mean by "flatter".) I usually do a screen print of the raw converter screen, paste it into a new Photoshop image, do a screen print of the comparison image,
post that into another Photoshop image, then display both images side-by-side. Finally, do a screen print of that. (Yes, it can be tedious:(


Even if I posted full size images I dare say everyone would have differing opinions on which images look better. ;)
This has tested my brain matter too much, I think I need to have a lie down!
Take one of these (https://www.google.com.au/search?q=bex&tbm=isch&tbo=u&source=univ&sa=X&ved=0CDIQsARqFQoTCLqP5fqz2sgCFWLopgodBEsEcA&biw=1920&bih=916#imgrc=9XPhumVM1rimPM%3A) first.


No need to reply!
You must mean AK:D

You don't need to read this.:D

[Aussieden]

Thanks "am" I think you have clarified matters for me. Time for another:beer_mug:

[Warbler]

I don't suppose you blokes have clicked on the drop down in ACR to change the way your RAW files preview, have you? Give it a try.

120970

[Warbler]

Oh, there's this drop-down menu too. This is for picture styles to use the Canon vernacular.


120979

[ameerat42]

I certainly haven't done this. - I don't have/use ACR:o
However, when I ever get to do so...

[Warbler]

Aren't you the proud owner of CS2, AM? It's been a while, so forgive me if I'm wrong, but I think it has an earlier version of ACR. The tabs may not look the same and maybe it doesn't have as many options. I guess what I'm trying to convey is that if your RAW previews, and hence your conversions in Adobe don't match what you're used to in Nikon's software, then maybe you aren't using the correct settings.

[ameerat42]

Re ACR: You are right, but I don't have any experience with it. When I used to dabble with an older Nikon camera - trying to help its owner - I used
View NX. Several years have elapsed since then and my recollection of the Nikon SW has evaporated.

- - - Updated - - -

Ie, I checked and ACR is there.

[Warbler]

The only fly in the ointment for using ACR in CS2 will be that by now very few cameras will be supported by by Adobe's version of ACR that works with CS2. They used to not make ACR backwards compatible to earlier versions. Call me cynical, but Adobe wanted to make you upgrade to every new version of CS they released. They still do, but as a concession to CS6 owners they made more recent versions backwards compatible until the most recent which now only CC users can access. The subscription models does what they couldn't do previously. It keeps the dollars rolling in.

[arthurking83]

.....

My understanding is all RAW files have some post processing applied or you would not be able to view them. Therefore there's no such thing as an unprocessed RAW file. Is that correct?

With regard to Olympus RAW files and Olympus Viewer 3 my understanding is that it uses the in camera settings selected at the time the image was shot as the starting point which then represents the preview image on your screen. .....

Yes! for sure.

Obviously I have no experience with Olympus's software, but I'll explain Nikon's basic software a little, and it may translate to Olympus's a bit.

(Almost for sure: but I don't know!!) OV3 will be displaying the jpg within the raw file anyhow, rather than the raw file data itself. This is how most of Nikon's software works(or at least used to work).
But Nikon changed their free software(editors) and they changed more stuff about how they now operate.
With the old ViewNX2, you had the option to view either the jpg preview file within the raw file, or switch to render the raw file directly.
Basically what that meant was that you viewed a jpg file on your screen, or a tiff file(with the raw file enabled). The raw option made the program a little slower.
What you view on the camera's preview screen is a jpg version of the raw file(even if you don't use the jpg option in camera).
The raw file contains at least 1 jpg file of full resolution. Some other raw file types contain more(1 or 2 more) jpgs files for other uses(eg. thumbnail images, etc).

There will always be some processing of the raw file, even from within thirdparty editors, like Lr ... et al
They have too, otherwise you won't see an image as such.

Back onto the Nikon software topic tho:
With the older software(ViewNX2), when you edited a raw file, it added the edit(eg picture style, exposure compensation, etc) into an area of the raw file that was set up for this.
In doing the edits this way, what happened to the raw file was that the internal jpg image(the full sized preview image) was also updated to reflect the edit change.
So, if I viewed an image in another raw file capable program eg. Fastone Viewer, Windows file explorer, etc .... or even Lr to a limited degree it basically displayed the preview jpg file when showing the image .. not the raw file data directly.
(Still on this bit of info tho, with Lightroom, when you initially opened Lr and imported the collection of images prior to any editing done on them by lightroom, in the thumbnail list, you can visually see that jpg preview file being used to display the thumbnail of the image. That image is not the larger of the jpg images within the raw file, but a smaller thumbnail version of the jpg instead.
Once you then start editing that raw file tho, Lr then begins to use it's own preview file.)
So the older Nikon software physically updated the embedded jpg preview file and it maintained a level of consistency when viewing the raw file across various software.
Now tho, Nikon started to use the external sidecar file system for editing the raw files.
When you edit a raw file, instead of updating the raw file in in a physical sense, it does so in a virtual sense with the sidecar file system.
So only it knows what edit you've done to the raw file. Viewing that raw file in another program only displays the original raw file preview image.

So while it's true that the raw file is processed, in reality when we say it's straight of camera, it really isn't processed as such. But yes it is. (it's confusing).

So even that raw file viewed in ACR, or whatever other thirdparty program that may look flat to you, it really isn't as flat as you think.
The reason for all that info immediately above is relevant now, because while you think Lr or ACR, or DXO or C1's version of the raw file looks flat, the raw file itself is set as it is in camera!

So for example(using your Olympus environment): if you shot an image with the most wild contrasty/vividly saturated settings and opened the image in OV3, you obviously open the raw file as shot in camera. If you open the image in Windows program capable of previewing that raw file(with the Olympus raw codec installed in Windows) .. you will see the same image as you see in OV3.
If you go to open the image in Lightroom(which uses ACR), prior to actually opening the raw file(so at the thumnail preview stage), you should see the thumbnail as the image was shot in camera.
Once you open the raw file, Lr's image rendering process takes over and it uses it's own internal system for displaying the raw file. BUT(once you have done this in Lr), if you open the image again in any of the above software mentioned(FSViewer, OV3, Windows Explorer, etc) .. it's still rendered as shot in camera.

[Warbler]

(Still on this bit of info tho, with Lightroom, when you initially opened Lr and imported the collection of images prior to any editing done on them by lightroom, in the thumbnail list, you can visually see that jpg preview file being used to display the thumbnail of the image. That image is not the larger of the jpg images within the raw (http://www.ausphotography.net.au/forum/showlibrary.php?title=New_To_Photography:Appendix_B_-_Raw_and_White_balance) file, but a smaller thumbnail version of the jpg instead.
Once you then start editing that raw (http://www.ausphotography.net.au/forum/showlibrary.php?title=New_To_Photography:Appendix_B_-_Raw_and_White_balance) file tho, Lr then begins to use it's own preview file.)

The bit I’ve bolded from this rather wordy tome goes back to the point I was making. What you see previewed will depend on the settings you have in your viewing software. You need to make sure you know your viewing software and how it works.

In Bridge there is another trap for young players. In the Default Image Settings bit, uncheck the “Apply auto tone adjustments” box too if you want total control.



120982