DxOMark results are...

[etherial]

... a waste of time! (IMO!)

I always had my suspicions about these DxOMark tests and have read various people talking them up (recently Nikon fan-boys about the D800 score) and also people saying they were rubbish. (Note this is NOT a brand war thread!)

Well today I've gained some experience to make my mind up. I normally shoot with a 7D but recently I forgot to take it to a herding event and borrowed a Nikon D5100 to take some shots. (the best camera you have is the one in your hand and at least I got some shots!!) I've just edited those photos and I'm blown away by how washed out and poor the photos were. They all required almost a 1 stop bump in exposure and a strong contrast tone curve to be applied and then further pushing with highlight and shadow sliders. I really struggled with getting a decent looking end result without breaking those pixels. I am amazed at just how poor the files were out of the camera. Examples are below, sadly I don't have a 7D photo from the same conditions to compare, but I have shot there with my 7D on different days and been happy with the results. Of course the lens will have a significant bearing on this as well, but again, my experience with my old 450D and kit lens was far better that these. (Again I'm not trying to bag the 5100, just trying to illustrate the point about the test scores).

So out of interest I jumped onto the DxO scores for the 7D and the D5100 and was amazed to see that the D5100 scores massively higher than the 7D. I can honestly say that these results are total bollocks! The results should be the exact opposite in my opinion. These results may give them a number in a lab but in a real life situation, it couldn't be further to the truth. To take it a step further I compared the D5100 to the new Canon 5D3. OMG the D5100 scores just one point lower!! :eek: Now this of course is a very unscientific test, and the glass will make a difference, but the comparison of these scores is simply staggering.

Comparison at DxOMark here. (http://www.dxomark.com/index.php/Cameras/Compare-Camera-Sensors/Compare-cameras-side-by-side/(appareil1)/795%7C0/(brand)/Canon/(appareil2)/698%7C0/(brand2)/Nikon/(appareil3)/619%7C0/(brand3)/Canon)

I'm blown away by this. I was looking at these results for the 5D3 a couple of weeks ago trying to get a gauge of what sort of difference I would see from my 7D to the 5D3 I'm thinking of upgrading to. After seeing this today, I will be deleting the DxOMark from my bookmarks. It is useless. The real world is where it matters.


This is one example from the day, all the photos had very similar setting changes in Lightroom. This thread could easily double as a "why you MUST post process your photographs" thread! :D

Original jpg straight out of the camera cropped mildly to improve composition, shot in Aperture priority, ISO400 f5.6 1/800s, pattern metering. The histogram is bunched up in the middle and no-where near the blacks or whites.

http://etherialdesigns.smugmug.com/photos/i-fHgHKvt/0/L/i-fHgHKvt-L.jpg


Lightroom edited, Strong Tone curve applied, Exposure +0.9, Contrast -5, Highlights -78, Shadows -54, whites +2, blacks -71, clarity +25, vibrance +5, saturation -15, and some sharpening. I'm still not happy with the hue of the red, but I think the result is reasonable. Overall I think I've managed to push this image as far as I'd dare, I'm not real impressed with some of the banding in some areas that has resulted.

http://etherialdesigns.smugmug.com/photos/i-XZRcvJz/0/L/i-XZRcvJz-L.jpg

[Eberbachl]

I have extensive experience with Nikons and have even owned a D5100. You can very easily get outstanding images straight out of the camera.

If I went shooting with a borrowed 7D I'd probably produce sub-par images, but I wouldn't blame the camera as I know full well it's capable of stunning results.

:D

[etherial]

As I said this IS NOT a Nikon bashing thread. You have missed my point entirely.

[Lance B]

I think what DxOMark results are eluding to is the potential of the IQ that can be achieved from a specific camera, which is exactly what you have shown with your post processing.

Just because you have to work at an image from a camera doesn't mean that it is bad, nor that DxOMark's test results are wrong.

I think you are being a little oversimplistic with your test methodology.

[etherial]

I agree it is a rather unscientific test, but think about this; they rate the 5D3 and the D5100 only 1 point different, are you suggesting they are capable of similar results? I can't imagine getting an image straight out of the camera from a 5D3 like the first one above. If that were the case why bother getting better bodies, just invest in glass, which reinforces my thoughts that such ratings are a waste of time.

[Tommo1965]

i think you might have the camera set to standard image in Jpeg..that gives it a washed out look IMO..try Vivid next time...then you will get a image similar to the edited version of your shot ..all raw shots will have the washed out look too if you shoot with matrix metering and let the camera do the work...or exposure compensate for a better look straight from the camera


so as far as DXO..not sure that relevant in this case..more that fact that your not used to the Nikon and have it set different than what your used too

[ameerat42]

Etherial. Much as I hayte to poke fingers at people's experiences, this just sounds to me like an experience that totally failed to meet your expectations.

As you presented the situation, it doesn't prove anything. It's all anecdotal.

I don't give a pair of owl's hoots about DxO marks either, but I will say that what you presented is hardly on a par with whatever testing regime they (DXO-ers) based their conclusions on.

Much of your dissertation reads like a powerful negative reaction to something. What settings were on the camera? Standard everythings?

Basically, it's a different camera from what you're used to. I would say, expect some different results, and when you get them, try to interpret them a little more constructively than just the general waste of time that you have reported.

Please don't feel offended.
Am.

[Eberbachl]

As I said this IS NOT a Nikon bashing thread. You have missed my point entirely.

No, not at all... you're implying that the D5100 does not produce nice jpegs out of the camera.

The fact of the matter is that you have not adjusted it's settings correctly to produce the sort of image you require. It's simply a case of driver error. The D5100 can produce sharp contrasty and nicely saturated images with very little effort if it's set up correctly.

;)

[Eberbachl]

Additionally, the specific area in the DxOMark comparison where the D5100 really caned the 7D is in low light performance. Hardly a relevant factor in the image you posted. The D5100 beat it in dynamic range where it's done a good job here (plenty of detail in the shadows and highlights), and only slightly beat it in colour depth.

There's nothing in the image above that actually disputes DxOMark's findings.

[Tannin]

DXO has been a well-known very poor joke for a long, long time. Mic, I don't pay any attention to speak of to your bad experience with a D5100. I bet you that once I got to know it I could produce excellent results with it. What's more, I bet that you could too. Or any of us here. It may or may not be slightly better or worse than the equivalent Canon (a 550D, I guess) but any difference will be very small.

But when you point out that DXO has just produced yet another absolute shocker of a result, yet another "scientific" finding so steeped in absurdity that it was probably originally rejected by the Japanese Whaling Publicity Office on the grounds that it was too ridiculous .... well, why would anyone be surprised?

DXO: the software which "proved" that the D200 had better high-ISO noise ability than a 5D Mark 1. :lol:

The DXO heritage: it started out completely useless and has maintained the tradition perfectly right up to the present day.

[Eberbachl]

DXO has been a well-known very poor joke for a long, long time. Mic, I don't pay any attention to speak of to your bad experience with a D5100. I bet you that once I got to know it I could produce excellent results with it. What's more, I bet that you could too. Or any of us here. It may or may not be slightly better or worse than the equivalent Canon (a 550D, I guess) but any difference will be very small.

But when you point out that DXO has just produced yet another absolute shocker of a result, yet another "scientific" finding so steeped in absurdity that it was probably originally rejected by the Japanese Whaling Publicity Office on the grounds that it was too ridiculous .... well, why would anyone be surprised?

DXO: the software which "proved" that the D200 had better high-ISO noise ability than a 5D Mark 1. :lol:

The DXO heritage: it started out completely useless and has maintained the tradition perfectly right up to the present day.

I don't want to be seen to be defending DxOMark here, as I never look at their scores and know little about their testing methodology.

Your comment that they rate the D200 higher than the 5D MkI in terms of low light performance intrigued me however as it seems ridiculous that they would do so, so I looked up the scores.

They rate the Nikon D200 at 583 ISO

They rate the 5D MkI at 1368 ISO (2.3 times better)

Seems to me that they rate the 5D MkI significantly higher (as they should) than the D200 in terms of low light performance.

[etherial]

OK well for those that see this as a Nikon bashing then fair enough, lets make it one!

Regardless of how good or bad the sensor is, if I as an reasonably experienced amateur photographer can't pick up this camera and get good shots out of the camera in a couple of hours, then the camera is poor (again IMO!). Furthermore, no matter how good a sensor is supposed to be if the camera can't process them to give me a reasonable result on basic settings without digging through menus or going away from standard picture styles, in my mind it is a poor camera. There will be many that will be disappointed with the results if that is a representative example of what to expect from it. I am not typically a brand fan-boy and I respect Nikon for producing some great products that I've considered strongly for various reasons.

I looked back through my Canon 450D photos with a kit lens and out of the camera jpgs are much more pleasing and easier to work with than these were. Look back at my photo of them month winner from years ago, that photo was taken with the 450 and kit lens on my very first day with it when I knew nothing! And the result out of the camera was pretty good. If these photos above are representative of what a D5100 produces, well I made the right choice way back then.

At the end of the day, my point still stands, lab tests like this that put the likes of a 5D3 on a overall par with a 5100 or in this case put a 5100 well ahead of a 7D are a waste of time because as I've demonstrated either through fault of the camera, or fault of my ability to use the camera in the real world the results will be wildly different. Even if you choose to believe the results regarding the sensor, there is much more to a camera and the resultant images it produces than just the sensor.

Just my opinion! Flame suit on! ;)

[Eberbachl]

No body is making it a Nikon bashing thread... simply pointing out that if you don't succeed in adjusting the camera to get the results you wanted (that which the camera is quite capable of) it doesn't mean that the DxO Mark testing results are any less accurate. Only that you haven't optimised the camera correctly to suit your needs.

The discussion is brand neutral in my opinion, the same would go for any camera.

;)

[Eberbachl]

Regarding the 5D MkIII Vs D5100 comparision, they say that the D5100 has a higher dynamic range, the 5D MkIII has much better low light ability, and the 5D MkIII slightly better colour depth. They're not saying anything much else about the cameras other than that. Do you dispute those specific results?

[etherial]

Regarding the 5D MkIII Vs D5100 comparision, they say that the D5100 has a higher dynamic range, the 5D MkIII has much better low light ability, and the 5D MkIII slightly better colour depth. They're not saying anything much else about the cameras other than that. Do you dispute those specific results?

I've not used a 5D3 so I can't really comment on the specifics of each rating. But the fact that overall they are rated only 1 point different tells me something is wrong with either their testing method or their score weightings. What I would say though is that they rate the 5100 better than the 7D by some margin in all areas which based on my experience I find staggering.

[Eberbachl]

I agree with the notion that the image out of the D5100 above is sub-par, but none of the issues with it relate to colour depth, dynamic range, or low light performance (DxO Mark's tests) so I wonder why based on that you would dispute their findings about the 7D Vs the D5100.

The 7D and D5100 cameras are both beaut pieces of gear and capable of excellent results when driven well. I know I wouldn't be able to drive a 7D nearly as well as a D5100 so my images from a D5100 are likely to be far superior than my images from a 7D. That in itself doesn't mean much except than I'm better at driving a Nikon than a Canon.

;)

[etherial]

I agree with the notion that the image out of the D5100 above is sub-par, but none of the issues with it relate to colour depth, dynamic range, or low light performance (DxO Mark's tests) so I wonder why based on that you would dispute their findings about the 7D Vs the D5100.

Interesting. So if the image is poor, but for reasons not related to those that they test, then my only conclusion that there is something lacking from their testing regime which reinforces the title of my thread?




The 7D and D5100 cameras are both beaut pieces of gear and capable of excellent results when driven well. I know I wouldn't be able to drive a 7D nearly as well as a D5100 so my images from a D5100 are likely to be far superior than my images from a 7D. That in itself doesn't mean much except than I'm better at driving a Nikon than a Canon.

;)

I bet you I could hand you my 7D and you would get good results from it easily. Hell I stick it in Aperture priority, auto ISO and give it to my wife who's photography abilities are surpassed by her iPhone and she still manages to take reasonable photos with it! :D

[Eberbachl]

It seemed to me that you were arguing that DxOMark's results were not accurate based on the image you presented. My point was that your image didn't do anything to dispute DxOMark's findings. Are DxOMark's tests useful? Maybe not, maybe so. I think it depends on whether you understand what they're testing and whether or not those tests are useful to you. Personally I never look at them. I'd rather look at photos than lab tests. I just haven't found anything here to dispute their accuracy - that's all.

;)

Getting great photos from any camera depends on your ability to adjust the settings to taste. I don't know what any of the jpeg processing controls in the D5100 you used were set to, but I agree they were bland. I used to get great jpegs out of my D5100, and my wife could too.

;)

[Tommo1965]

OK well for those that see this as a Nikon bashing then fair enough, lets make it one!

Regardless of how good or bad the sensor is, if I as an reasonably experienced amateur photographer can't pick up this camera and get good shots out of the camera in a couple of hours, then the camera is poor (again IMO!). Furthermore, no matter how good a sensor is supposed to be if the camera can't process them to give me a reasonable result on basic settings without digging through menus or going away from standard picture styles, in my mind it is a poor camera. There will be many that will be disappointed with the results if that is a representative example of what to expect from it. I am not typically a brand fan-boy and I respect Nikon for producing some great products that I've considered strongly for various reasons.

I looked back through my Canon 450D photos with a kit lens and out of the camera jpgs are much more pleasing and easier to work with than these were. Look back at my photo of them month winner from years ago, that photo was taken with the 450 and kit lens on my very first day with it when I knew nothing! And the result out of the camera was pretty good. If these photos above are representative of what a D5100 produces, well I made the right choice way back then.

At the end of the day, my point still stands, lab tests like this that put the likes of a 5D3 on a overall par with a 5100 or in this case put a 5100 well ahead of a 7D are a waste of time because as I've demonstrated either through fault of the camera, or fault of my ability to use the camera in the real world the results will be wildly different. Even if you choose to believe the results regarding the sensor, there is much more to a camera and the resultant images it produces than just the sensor.

Just my opinion! Flame suit on! ;)

LMAO..sounds like your the only one here making it a Nikon bashing thread...and thats what you really want.....to say Canon is better....LOL::lol:..and for you it probably is

[etherial]

LOL, I didn't want to make it one, my point really was directed more at the DxOMark ratings, but I think everyone took it as me having a dip at the 5100.

So yes, while it wasn't the point of the thread as I don't generally like to get into brand war discussions as they are pointless, but you're right, I didn't like it! :D

[Lance B]

I agree it is a rather unscientific test, but think about this; they rate the 5D3 and the D5100 only 1 point different, are you suggesting they are capable of similar results? I can't imagine getting an image straight out of the camera from a 5D3 like the first one above. If that were the case why bother getting better bodies, just invest in glass, which reinforces my thoughts that such ratings are a waste of time.

No I am not suggesting that the 5D3 is only 1 point better than the D5100 overall. The 5D3 is far the better camera, what DxOMark measures is sensor performance, not camera performance, which is two different things. The 5D3 has much better AF and faster frame rates, better built, more comprehensive menus and customisability. This is whay the 5D3 is by far the better camera over the D5100.

It's like saying a WRX is better than a Porsche simply because it has more HP or whatever. It just ain't so.

[zollo]

It's well known that Nikon don't apply as much 'polish' to their in camera jpegs as canon do. Probably takes a bit of getting used to after the over saturated jpegs canon puts out.

Also I can't see/tell where the focus is in your example, its certainly not on the person, camera error too... Maybe.

If you really want to see bad jpegs from a camera use ISO 400 or above on the 7d ;-)

[I @ M]

Mic, I really think that you need to sit back, have a cuppa and look into the DXO methodology a little harder.

I'm not either leaping to the defence of or decrying the DXO results because I simply don't understand them and Lance ^ above is on the mark with his summation of the DXO tests.

DXO, simply put, don't produce any images in their testing. That means that lens IQ differences are a moot point. The actual image presented on a screen is also a moot point because the image output from the camera especially in jpeg form is heavily dependant on the processor in the camera and with their testing consisting of electrical input and output and mathematical formulae being applied only to the sensor and before the processor stage DXO are merely presenting numerical results.
Those results may or may not be able to be proven correct or otherwise but until someone agrees on a standard the results from DXO are representative of their and their alone, testing.

I think with your "testing" and then understanding of the published DXO results comparing the bodies that you have overlooked a few things.

With the DXO results, you have applied the wording "massively higher" and I don't see it that way at all. What does the difference between 80 and 66 represent? Is it a percentage, number of oranges in a kilogram or tonnes of wasted whale meat from the Japanese fleet between 2009 and 2012?
Looking beyond the final "score" and into the graphs and things that I can understand a little more like ev and iso I don't see massive differences.

As for your presented image being preventative of what a particular camera can produce and used as an example I'm afraid that it simply does not work.
Too many questions are needed to be asked about that shot before any credence can be paid to the performance of the sensor ( which after all is what the DXO tests are about ) and you need to answer them thoroughly to be able to substantiate any observations about either the end image quality or the performance of the sensor.

From the very start.

#1 What picture control was used in the camera at the time of the shot?
#2 What alterations were made to that picture control, either on purpose or accidentally by the previous user of the camera or by yourself?
#3 What white balance settings were used and what alterations to the tint were applied?
#4 If the D5100 has the facility within the menus, as other Nikon models do, to set an overall exposure compensation adjustment without displaying that compensation adjustment, was that enabled?

Looking at that image and applying some very minor adjustments in the software that is supplied with that camera suggest to me that it was taken in either standard picture control with the contrast reduced ( even maybe no lens hood in place? ) or in the neutral picture control.

Either way, I don't see that image as representative of the mathematical testing of a sensor or as an indication of IQ from a particular camera.

[farmer_rob]

It seems to me that the DXO test results are technical evaluations of sensor performance, not of the secret sauce that Canon and Nikon use for converting raw sensor data to jpegs. (IIRC, DXO do tests on RAW files only.)

I think you have picked a difficult image - at least for NIkon cameras. It took me quite a lot of work to figure out the correct camera settings for black cattle and black dogs - the default nikon exposure calculations give you pictures like your first one, or (if you are not careful) pictures with black featureless masks and nicely exposed surrounds.

It may well be that the Canon exposure/processing calculations assess the scene "better" given the need to nicely expose the animals without washing out the rest of the scene.

These exposure and processing settings are not assessed by the DXO tests. Instead the tests just provide plenty of "canon" fodder for nikon fanboys :-)

Regards,
Rob

[ricktas]

I would ask. When you borrowed this D5100 did you do a complete reset of the camera? I see you initial photo as being SOOC, so what settings were used (not the ISO etc) but shooting in JPG lets you choose a huge range of things in the camera menu, like saturation, sharpening, etc. Did you reset these to default and then adjust them to how your 7D is setup for you?

Without doing a thorough scientific test where all settings are set to as similar as possible, and similar lenses used across cameras, any testing like this in the real world is going to have a negative effect on the results from one of the cameras compared to the other. We need to know ALL and EVERY setting in both cameras are the same, before we could even begin to discuss the variations. DXO do this, they setup the cameras under as similar environments as possible and then test them.

[ricktas]

You were also using a superzoom (55.0-300.0 mm f/4.5-5.6), basically a kit lens, not known for producing L quality results. There are way to many variables here to determine anything, either with your photos, or the DXO comparison.

[etherial]

Thanks guys for contributing to the debate, makes for interesting reading. :th3:

You are right, I didn't go through menus to check every setting, but knowing the owner of the camera, they use it on full auto all the time and hadn't had it very long so I can only assume that those settings were left alone. As I said from the top, this wasn't a scientific test, I didn't even set out to do a test, just borrowed the only camera I could get my hands on to take some pics of my boy working at the trial. All my points are anecdotal, but one thing it is, is real world experience.

In any case all the above from all of you supports my thoughts that there are way too many variables to consider to make these ratings useful to apply to real world situations. Sure if you want to drill down and analyse and understand every test in detail you might get something out of it, but from camera sensor to computer screen or print there are many more important factors that contribute to final image quality than the sensor and its DxOMark score. So my advice for the punters out there looking for a camera, don't take any notice of the scores, there are more important things to worry about when choosing your next camera (one being the lens!!). :)

[farmer_rob]

More importantly, did your boy work well?


Regards,
Rob

[etherial]

Yes Rob he did pretty well. He's almost 10 and only been trialling for a couple of years. The cows were a bit of a challenge for him, they are a bit more stubborn and he failed on those runs but he tried hard. He did some intermediate sheep runs in the afternoon and passed one to get another title so it was a good day. More photos here (http://etherialdesigns.smugmug.com/Etherial-Border-Collies/Our-Dogs/Gr-Ch-Etherial-Chase-the-Ace/9196647_vvN2RL) if you're interested. ;)

[ricktas]

You are right, I didn't go through menus to check every setting, but knowing the owner of the camera, they use it on full auto all the time and hadn't had it very long so I can only assume that those settings were left alone.

HEHE. When you get a new camera, most people take a few shots then start 'playing' with the menu's. It is so easy to adjust something (even without realising) and then forget you have done it.

[arthurking83]

I think Rick is starting to hit on the point here Mic.

You have to remember something here too, Nikon's D7000, Pentax's K5 all have very similar DxO scores for a reason .. they all use this same sensor!!

Where the D7000 and D5100 only score 1 point less than the 5DmkIII, the K5 scores 1 point higher than the MkIII.
The pattern repeats itself due to the fact that Pentax and Nikon have eeked out everything they can from the Sony based sensor.
If you also place Sony into the scoring mix, again their camera's that also use this same 16Mp sensor also score well too!

I think the major point here is not that DxO is bollocks .. but that it's bollocks for your situation!!

You don't use DxO's software, so their results are useless for your purposes.

If Adobe produced some testing procedure that was similar to DxO's, using their software, then you would be singing their praises being so perfectly spot on.

You're comparing apples to cheese here, due to your workflow method.
You're reading on how DxO software evaluates the raw files, but using Adobe software to try to achieve the same results!

I can tell 'ya straight up, I can't the same look from my raw files using LightRoom, as I get using Nikon software(and this is the free Nikon software too boot!).
The free Nikon ViewNX2 always gives me nicer looking images instantly if I've got the capture right on.


Now the other issue I can see that may have tainted your unscientific results, is that by default, the D5100 is set to use Active D-Lighting, and it may be set to auto(this is how I found my brother's D5100! .... can't really remember, but ADL is not an ideal feature to use all the time. It's handy for specific conditions, but when conditions are such that it's not required, ADL will set your images to look flat and washed out.
I reckon this is what may have happened. Go to the Camera Menu item(Shooting Menu) and scroll down till you find Active D-Lighting. Any setting other than Off will produce slightly flat looking results where the histogram is all bunched up in the middle!!

On a raw file, Lightroom won't recognise the ADL setting you set, but the jpg will be affected so.

Also, Lightroom doesn't render the Nikon raw files as well as Nikon's software does. An exposure adjustment is ALWAYS required in LR, where it's not in Nikon's software, and this can be either up or down in Ev. I's very rare(for me) to have found a Nikon raw file that is exposed the same way when comparing Nikon's software with LR.
Again, as it's a raw file, it's simply a difference of opinion(between Nikon-Adobe-DxO-and others) in the way the file should be rendered.

While I'm not a fanboi of DxO's, I still respect their results, as it's most probably very pertinent when using their software for evaluating or comparing various cameras.
But if you're only using other software, then the recorded results may not coincide with the theory.


As a quick test: If you have access to FSViewer you can use this to compare how the images look between software.
Open any raw image in both FSViewer and Lightroom, and notice the difference in rendering between the two software.
It'll be blindingly obvious if Active D-Lighting is enabled.

[etherial]

Thanks Arthur, all very valid points... which reinforce my view that high DxO scores will not lead to happiness.:D

[arthurking83]

..... which reinforce my view that high DxO scores will not lead to happiness.:D

Possibly not, if you're preferred raw converter is not DxO's own!!

This is why I mentioned FSViewer.

With FSViewer, there are options to render the embedded preview file(the jpg in the raw file), and also to view the raw file via the raw file data(slower image rendering).
This clearly shows the difference between raw file viewers/converters more aptly than most other software.

eg, in Nikon raw files, the embedded preview file is almost exactly as Nikon software will render it, too, which is as the camera captures it .. so Nikon software will render the raw file as you saw it in the camera, whereas third party software is always going to render the file differently.
So it goes that DxO renders raw files differently to how Adobe's raw converter(ACR) will render the file.
This is also complicated by the different manner in which each manufacturer's raw files are handled by the various software too.

maybe ACR has a better understanding of how to interpret Canon raw files than it does for Nikon raw files.

Basically it's as simple as this: DxO's rankings are meaningless as a single source of information without the use of their software.
Unless we find ourselves in a situation where all the various manufacturers use DxO's raw conversion engine to display and edit images, DxO's reviews are only one guide amongst many others on the topic of relative camera performances.

So, with all things information related, you use DxO's reviews as part of a whole swag of reference reference material for determining how a camera performs against another.
For Adobe users, DPR(who only seem to use ACR as their main raw file conversion program) would be a better source of info.

FWIW: if you took FSViewer's interpretation of Nikon's D800 raw files as the only source of information as to how the raw files look, and hence how the camera performs ... you'd be seriously disappointed!
Great software and I think it's the best at what it does, but I never use it for raw file conversion.

I can't remember DxO all that well other than it never impressed me much(long time ago), Bibble(now After Shot Pro) was fast but awkward, and I ended up with LR for any of my non Nikon raw file handling.

[patrickv]

Damn, I own a 7D and I don't even know what its JPEG files look like ...

[arthurking83]

Damn, I own a 7D and I don't even know what its JPEG files look like ...

They either look like whatever you want them to look like, or they look like what's displayed on the review screen!

You're not seriously going to tell us you've never reviewed an image on the review screen, are you?

[agb]

It's not only the sensor scores that give a bit of heartache. Try having a look at the scores for your lenses.
Especially Canon lenses.
According to DXO a score of 10 is required for a lens to be rated as capable of producing an excellent 8"x12" print on a home printer.
The 70-300mm f 4-5.6 L lens on the 7D gets a rating of ...............you guessed it 10.
How does that relate to the MTF charts for the lens that Canon produce which shows that this lens is a really good lens.
http://www.usa.canon.com/cusa/consumer/products/cameras/ef_lens_lineup/ef_70_300mm_f_4_5_6l_is_usm
Surely it can be good enough to produce better than a 8x12 print?
Users of the 70-300 have rated it as good as the 70-200 f4L IS which is also supposed to be a good lens, but it also only gets a 10. No telephoto lens on the 7D gets a rating good enough to print a 16x24, and most of the telephoto lenses would not give a print bigger than a postcard.
http://dancarrphotography.com/blog/2010/11/17/canon-70-300-f4-5-6-l-is-review-vs-70-200-f4-l-is/

[Tannin]

Your comment that they rate the D200 higher than the 5D MkI in terms of low light performance intrigued me however as it seems ridiculous that they would do so, so I looked up the scores. Seems to me that they rate the 5D MkI significantly higher (as they should) than the D200 in terms of low light performance.

Yeah. they do now, 'cause that was one of their stupidest mistakes and they had no choice but to fix it. But each new version of DXO seems to make a habit of throwing in a different howler. Etherial started this thread with yet another one. (I should mention that I'm going on memory for my model numbers, it may have been some other equally ridiculous finding that first drew my attention to DXO all those years ago. But it doesn't matter in any case, there are plenty of other howlers. The sensible thing to do with DXO is simply ignore it in favour of other methodologies that produce more rational results.)

[Eberbachl]

Remembering that they're rating sensors and not cameras, most of the results I've read since stumbling across this thread have seemed pretty credible thus far. If you have any current examples of howlers I'd love to read them.

;)

[Tannin]

^ Etherial started the thread with one. Given that we already know the program makes huge mistakes, what would be the point ion looking for more examples? I've got 1001 better things to do with my time, but you could always go looking for yourself if you are bored enough. Not me. Life is too short!

[JM Tran]

^ Etherial started the thread with one. Given that we already know the program makes huge mistakes, what would be the point ion looking for more examples? I've got 1001 better things to do with my time, but you could always go looking for yourself if you are bored enough. Not me. Life is too short!


Im with you there, DxO is rubbish as established. Less debating numbers and decimal points, and more photo taking needed! :D

[etherial]

Another "interesting" result I noticed last week (and just remembered to post) was their ratings of the Canon 70-200f2.8 IS II against the older version of the lens. The scoring system they use rates the older version better for all except one parameter, yet in their "user case scores" they give the new one more stars.

http://www.dxomark.com/index.php/Lenses/Compare-Camera-Lenses/Compare-lenses/(lens1)/408/(brand)/Canon/(camera1)/436/(lens2)/222/(brand2)/Canon/(camera2)/436

Then they write an article comparing the two:
http://www.dxomark.com/index.php/News/DxOMark-news/Canon-EF-70-200mm-f-2.8L-IS-II-USM-measurements-and-review

I wonder what Canon thinks of such results? Hard to imagine they would spend all that money improving a lens only for it to go backwards? And just about every other review I've read on the version 2 says it is an improvement over the (allready excellent) version 1.

[arthurking83]

If you read all(but one) of the reviews of the Nikon 70-200/2.8 VR lens comparions, everyone agrees that the VRII version is sharper, yet Photozone's 'numbers' clearly show that the VRII version is actually sharper at 200mm(in ultimate raw numbers).
Look deeper into this strange assessment tho and the difference is trivial(a single digit or two) .. but it is still technically sharper.
The newer VRII lens is better overall in all other aspects, other than raw values of an almost meaning test procedure.

It can be normal for a newer lens to be 'worse' than the predecessor lens it's supposedly replacing, but there are other gotcha's to be weary of in this situation too.

Sample variation!!

We all know it exists, and maybe DxO got a dud.. or didn't sample many lenses .... or if they did use many samples, they just happened to luck right out and get all duds!!

I remember that when PZ finally posted their results for the 70-200 VR(1st version) way back when, they made a note that explained that there was a delay in the posting of the results, and used two samples of the lens because the first results were quite average(for such a high end lens).
I'm sure they had a link to post the differences, but can't see that now.
The second sample provided figures, graphs and numerical values that look more like what a super expensive pro lens should produce!


DxO's scoring system for lens tests is ridiculous to say the least!!
Whether they're accurate or not is not really part of the problem as I see it,

That they think they can assess a diverse range of lenses into a lowest common denominator manner with a single figure score is simply stupid!

[Bennymiata]

I think that one thing that must be taken into consideration, DXO are French and anyone who has dealt with French companies will know what I mean.

[arthurking83]

..... DXO are French and anyone who has dealt with French companies will know what I mean.

C'est ... LOL! :D

"we are right, and we fart in you silly English knigget's general direction!" :D



But it also must be noted, that it was them that invented this photography thing too.
.... and not only did they invent it, their government of the day purchased the patents and declared the earliest photography processes patent free and for everyone to use as they wished.
I can't imagine that having happened on another (unnamed)continent on the other side of the world tho!

[etherial]

I think that one thing that must be taken into consideration, DXO are French and anyone who has dealt with French companies will know what I mean.

I work for a French company, and I know exactly what you mean! :p