DXOMark how good.

[martycon]

When considering purchasing lens or body, do you use DxOMARK or similar objective measurement publications to help you arrive at a decision? Are there others that you can recommend.
I am impressed with DxOMARK, but am not sure whether my trust is misplaced.

[ricktas]

For lenses I tend to go by what others think (real life users) and looking at some of their photos (the raw files), rather than look at review websites. To many of these sites are now owned by those with on interest in marketing products that their reviews could be biaise. Take DPReview (owned by Amazon).

Whilst DxO is one of the better ones from what I can ascertain, I prefer the advice of fellow photographers.

camerawise it is a bit more difficult when you are set with a specific brand, as most are. I tend to look within my chosen brand (Nikon) at their offerings rather than consider a review site. Though when I bought my D800 I did look at the DPReview umm review to see what they had to say, but I doubt it would have swayed me, unless they pointed out a huge issue. And that issue was backed up by other reviewers, elsewhere.

Certainly, I think review sites are good if you are going into something (not necessarily photography), fresh and are accumulating knowledge to make a decision on what to buy.

[arthurking83]

IN reality I think DXO's camera ratings are only really relevant if you use their software!

That is, they rate a cameras ability to render an image in a certain way based on the results achieved by their software!

A cameras ability to render an image in a given manner could be vastly different using other image software(eg. the manufacturers software!) to how DXO's Optics Pro can.

This is not to say that DXO's ratings are immediately rendered useless .. they most certainly aren't.
It's just not the definitive answer. It's one source of data in a sea of other useful data!

Once you have a short list of what things you want the camera to be able to achieve for you, searching all relevant info about it to determine if it's the right piece of equipment seems only natural.

That's how I use it.

For example(in my situation).
I wanted a full frame Nikon camera. At the time I made that choice, to actually spend the money .. not the initial desire to get there! .. I had a few choices.

1. D700, was out for about years by then and was the catalyst fro my desire to go FF.
2. D800 then came out(2012) and increased the level of this desire.(problems started to surface as I finally got the money together, and .. )
3. D600 then came to market.

Searched lots of info about all the specifics I wanted. Video(capability) was a major factor in any new camera I was going to acquire. My old video recorder seemed anaemic by comparison to what was now possible.
So that removed the D700 from the list(even tho this camera was responsible for 4 years of sleepless nights .. and I@M allowing me to play with his D700 not helping! :p).
Downloaded many raw files from all three cameras to see for myself what was possible and hence capable .. and more importantly worked on my PC. 100Mb raw files apparently made your PC grind to a halt!
Using Nikon's software, I didn't see the same high ISO results that DXO did.
At insane high ISO(what they refer to as sports low light ISO) .. they rated the D600 and D800E as the (basically)same.
With all the sample raw files I downloaded (I think I still have some .. idiot I'll have to re-download many again to compare ... again) .. I found that at very high ISOs(eg. maybe at ISO6400 but definitely over ISO6400) .. not only did the D600 look cleaner with all NR cancelled out, but it looked cleaner after both raw files were reduced to the same normally viewed converted file.
This was only up to a point obviously where they seemed to render about the same (noise level) if the two images were sized up to the D600's native pixel dimensions.
ie. 100% view for the D600 file and about 80-ish% pixel size for the D800E file.
These sizes are not commonly reproduced, so I wanted to see them at say 3000x2000 pixels(possibly for a 30" print) to compare against what I could already get from my D70s and D300 at the time.
I've had two 30" prints from both the D300 and D70s(at base ISOs) and they both looked pretty much flawless .. more than acceptable.
All I wanted to see was what was possible if the situation called for 'extreme' shooting conditions.

So the D600 became a real contender .. and I could have been 99.99% ready to pull the trigger on getting one(back in '12).
Online prices were pretty high compared to some of the prices from established stores .. so I went in to check out the D600.
I almost knew that it wouldn't be the camera for me simply due to it's body form factor.
So even tho it cost me just a bit over $1K more .. I ended up with the D800E, even tho I'd have preferred the images from the D600(if the need ever arose).
There wasn't enough in it in terms of image quality difference, but the difference in body form factor(for me) was immense .. I needed the larger grip moulded better to fit my hand size/shape.
(I carry my camera in hand, and prefer not to use shoulder/neck straps).

So DXO was one source of info for me. Using as many raw files from each respective camera in my preferred image software was an even more important source of info for me tho.
The camera store, that is the physical aspect of how does it feel in my hand was the most important aspect tho.


I always check out what's happening in the camera market, as it evolves quickly .. one day brand A's camera B is the camera to get .. next day brand B's camera A is the one to get.

Like I said tho .. this doesn't diminish DXO's sensor ratings in any way. It's just that for me they weren't relevant for what I possibly wanted from the camera .. which BTW was to do a bit more astro imaging as part of something else to do.

As for other sources of comparison information .. I can't think of any ATM .. but unless you're an early adopter, I suggest to download (if you can) any files you think may be relevant to your situation.
That is, look for raw files if you shoot raw mode only .. but just download jpg files if you are a jpg only shooter!
Don't waste your time looking into irrelevant info.
Both DPR and Imaging Resource are two easy to remember sites where you can download sample files of various formats.


BTW. Another thing to be aware(or weary) of is as Rick said .. general user information.
Whilst general advice from fellow photographers is invaluable, be aware that it my be useless as well!

While I was deciding on which camera to get(D600 or D800), I was also aware of the possibility of the D800E's high probability of producing moire(colour artifacts).
I wanted to avoid this situation as the D70s used to do this(due to a weak AA filter) .. and so when you got it, it was ugly and (back then) hard to remove(hard as in it needed more processing work).
In the end I had no choice as the camera shop only had one D800E left in stock .. and as I wanted it NOW .. and not in a week or two AGAIN! :rolleyes: .. I thought to myself stuff it .. I'll put up with the more .. simply for the sake of finally getting some sleep :D
Nah!.... seriously, I downloaded some comparison raw files of identical scenes from both a D800E and D800 using the same lens same aperture .. etc.
While the D800E did produce moire where the D800 didn't, it wasn't as reported by others as a problem.
You have to be able to trust the person offering advice .. in that they are offering good advice in general, or good advice for you.
Hope that they're not just offering 'advice', which may not be relevant to you .. or even just plain old jibberish(carp!!).

From my experience in quite a lot of photography related sites .. AP offers both good and bad, which is good because you will get a good balance.
The garbage I've read on other sites(I won't name them again! :D) is well moderated or exposed for the junk that it is here on AP.

Note, that we will all have our opinions and preferences, as everyone does on any site where opinions and thoughts are sought.
But the major reason I've been a member of AP for as long as I have is that it's a very well balanced site to allow the possibility of multiple sides to every discussion.

[paulheath]

i tend you use the tried and trusted" if my wife lets me buy it" approach .. hasn't failed me yet :lol:

[Lance B]

I believe that DXO Mark is a reasonable place to start and their lens ratings seem to actually correspond with my findings of my Nikon lenses that they have tested, and I have done alot of photography with these lenses and have therefore got to know them very well. I also believe their sensor ratings are a reasonably good indicator. Other lens testing sites that seem to be quite accurate are Photozone, Ephotozine, Lens Rentals and SLR Gear. These are all places to do some research and get a feel for a lens, but it is also good to read people's thoughts from forums and some pro/advanced amateur users on line. If you are into Nikon, there is Brad Hill, who has an excellent website, Natural Art Images.

Just don't get too carried away with lens test chart results as they are usually only an indicator of what a lens can do as it is usually a test done at one camera to subject distance, not a range of distances which can affect the results either good or bad. Having said that, a lens that performs very well on a test chart is also likely to perform well at other distances, but it doesn't mean that it will perform quite as good or maybe not quite as bad. Usually, the difference is minimal anyway. Another thing is wide angle lenses which may not do well on a test chart due to their non flatness of field, whereas a test chart is flat, ie an extreme wide angle lens maybe sharp in the middle area but will possibly look less sharp on the edges and corners on a test chart simply because are designed for real world conditions and this means that they are designed to be sharp at different point at the edges and corners than the middle due to the fact that they can encompass the ground in front which will be at a different camera to subject distance than the horizon.

The thing is, when you have done a lot of research from many sites, a common feeling for the lens or camera should be apparent. It is rare that a lens tests well but is a dog in the field and visa versa.

[swifty]

I think there are a lot of useful info on DXOMark, the least of which is the final sensor score.
My feelings are that today, basically every large sensor is basically good enough. But DXOMark can give you useful info about each sensor's characteristics which may influence the way you shoot.
I use it not so much to pick a camera but to learn more about the sensor in the one I've picked.

I haven't used their lens resource much though.

[Hamster]

The way I look at it is, how would Adams' or Doisneau's gear score.

No the DxoMark doesn't matter one bit when I am choosing something (and this is from an engineer!). As others have said, real world practical feedback and a bunch of reviews from photographers who have used the gear is useful. Scores, not so much.

[martycon]

Thanks Paul, I love and value humour.

[J.davis]

So ken Rockwell is the go too man :lol:

[arthurking83]

i tend you use the tried and trusted" if my wife lets me buy it" approach ......


I used to have one of them too ..

when it came to the choice between having the bits I NEEDED, and the people I thought I wanted .... I went with the 'bits' ;)

haven't looked back since .. and my collection of bits I need is definitely approaching unmanageable (or maybe unimaginable!) levels :th3:
I found lenses that I totally forgot I had .. or can even remember why I got them in the first place. I'm sure it was for some obscure hair brained Frankenikonmonster idea I had some time ago.

[I @ M]

So ken Rockwell is the go too man :lol:

Go and wash your mouth out with lens cleaner young man. :p

[Brian500au]

I am in Rick's camp on this one.

I have invested in Canon gear early in my life so my search for updated equipment is confined to the Canon range. I rarely buy new gear, both from a cost saving point of view, but equally as much from a tried and tested point of view.

When I am looking for gear, I normally identify a need for the gear, i.e. do I want to expand my photography into macro work etc. Then I will identify which lenses I should look at to do the best job. Next I look at what I am willing to spend (which sometimes can change depending on my research). Next I will read up on the lens at sites like DxOMark, DPR, etc. Once I have narrowed my search down a little more I look for real life reviews on both photography forums and sites like B&H and Amazon.

Last but not least I decide what i want and then wait for a lens to become available on a used list somewhere. Quite often I can narrow my search to two lenses and but only buy the one that becomes available (and within my price range). I have done this with camera bodies, lenses, tripods and heads. I have found by doing this and buying used, if I need to move a piece of equipment on, then there is very little loss (I justify it as a rental charge).

I am not a early adopter of equipment, and I have found some great prices on camera bodies that are current models but at close to the end of their cycle (about to be replaced by a newer model). I have found in the Canon range the version I of the lens does not stop functioning just because version II was released last week, and once again becomes heavily discounted on the used market.

Just my 2c worth of how i go about acquiring camera equipment.

[MattNQ]

i tend you use the tried and trusted" if my wife lets me buy it" approach .. hasn't failed me yet :lol:

Hmm, I'd have no gear at all if I followed that approach - I must have married a faulty one... she doesn't view anything I want to buy as necessary :lol2:

------------------

In response to the broader question of the thread;

I agree with the general consensus that reviews by people that have used them in real life out in the field are what I place the most trust in.

Some get hung up on the placing of buttons or menus, but you get used to that. Sticking to the same brand means most stuff doesn't change much anyway.

It does need to be comfortable in hand and no reviewer can tell you what fits your hand best. Good camera stores don't mind if you have a play in store.

Ignore the fanboys who debate small differences in dynamic range, ISO noise, lens corner softness etc. Most of them spend more time behind the keyboard that behind the lens.

DXOmark does give you a rough idea of where lenses & bodies rank against each other, but I suspect most people would struggle to see the difference in real life between the higher end gear, especially if their photos only end up online and not printed large.

There are plenty of reviewers out there who give good real world reviews
Ken Rockwell mainly just good for a laugh, but I do read his reviews of old lenses if I see an old lens on ebay I am interested in.
Surprisingly those can have useful information and are more factual rather than his usual Rockwellian approach to most stuff.

[martycon]

Thanks Brian, my MO is very similar. My confidence in DxOMark is increasing.

[arthurking83]

LOL at Rockwell!

Strangely I sometimes read his reviews myself too.

If not just for the opportunity to discover a new faux pas somewhere on his website, then again just as another point of view to take into account.

Although why he wants a super high resolution 36Mp camera without an AA filter which implies "I want the best possible quality image here" .. and then shoot in jpg mode with all Picture Control settings taken to warp factor 12, Mr Scotty.

Back onto the topic of DXO's accuracy and or ability:

A long time ago, I became (personally) suspicious of their results.
Whist I still find them 'interesting' to a point, my suspicion then simply shifted focus from a possible commercial ulterior motive ... to their methods of analysis.

Quite simply. If you don't use their software, their results are more than likely.

Had they used an accepted or acceptable non interested third party raw converter routine(eg. DCRaw) and then analysed the resulting images to form a less obscure scoring system .. their results could be seen as an industry standard of some type.


The reason for my suspicions started back in about 2009 when Nikon introduced the D90.
Same sensor as the previously released D300, yet the D90(with it's supposedly lower pecking order status and using the same sensor, got higher low light ISO score than the D300.

Logic would say that for two cameras using the same technology and being marketed at the same time .. Nikon would make the higher priced item 'more capable' than the lower priced item.
Makes sense to be, otherwise who'd buy the 2x more expensive D300, if the D90 can do so much better?
That makes no sense to me if Nikon could or would do this.

So I went looking.

Not having a D90 myself, and even tho I had the D300, the only way to compare DXO's findings was to source raw files that were as close as possible to comparable between the two cameras.

So, having found many .. wayyy to many in the end, sample raw files and using Nikon's software(both VNX2 and CNX2 back then) ... I couldn't find this extra ability in the D90's high ISO settings.
I turned NR on and off and up and down .. and all I could see were images too identical to remember which came from which camera!

Initially my thoughts went with an idea that because the D90 has video capture ability and the D300 doesn't, maybe Nikon use some passive cooling hardware on the D90 and not on the D300.
if they did, could I do something with the D300 ... :D .. Nah! leave it. I was happy enough with the D300.

anyhow.

The only conclusion I could come too was that DXO are idiots(unlikely) .. or that they write their software in such a way that it varies from camera to camera even if the sensor in those different cameras are the same! :confused013

So I always liked DXO as an alternate source of info, but always viewed them with suspicion.

Then came the D4s/Df (what I call) debacle).

The Df is an older camera, works around a D4 sensor. D4s is an updated D4 .. ie. the same sensor.
When the D4s came to market, the expectation was that it would be the ISO 'king'.

Many Nikon haters would have smiled a huge wry smile when the DXO numbers came out. Df for some strange reason scores 'significantly' higher than the D4s in a low light ISO comparison.

Again! how can this be. D4 has video. Should have some way to keep the sensor cool during 29minutes of video capture. Cooler sensor = lower chroma noise.
This has been a passion/trait/holy grail for astro photographers for years! keep it cool get better images.

But the Df has no video and hence no need for a cooled sensor! So I got curious again and not having the ability to either acquire nor scientifically compare the two cameras, I went on the sample raw file search again.

I never questioned the quality of the sample raw files I find on the net, but I then did .. thinking maybe this was the problem.
After much searching I can't fault the sample raw files I find. Many from imaging resource, but not exclusively.

After opening and comparing both the Df and D4s files up to about ISO12800 they are identical. Df uses a bit less NR in camera(hidden settings), but with NR cancellled, they are identical anyhow.

But from ISO 25600, the D4s is better. This advantage is 1/. to be expected in real life 2/. very small 3/. in opposition to what DXO claim!

At ISO 51200 Df is obviously cleaner(with all NR off again).

Strange thing is, and without knowing every detail of DXO's processes .. DXO themselves have a graph for ISO which in effect contradicts their own scores for low light ISO.

At high ISO, the graph of SNR on a Df/D4s comparison shows a higher SNR for the D4s compared to the Df. Yet the Df is the low light king in terms of scores.

Like their lenses .. it just doesn't make any sense on how they come to the conclusion of their lens scores.

Lens 1 with a lower CA score(ie better CA compared to Lens 2), with a higher sharpness score, better vignetting score, better transmission score still scores an overall lower lens score than Lens 2 does.
It really makes no sense why the seemingly better lens(with better individual scores for each category can result in a lower overall score.

[Lance B]

The way I look at it is, how would Adams' or Doisneau's gear score.

No the DxoMark doesn't matter one bit when I am choosing something (and this is from an engineer!). As others have said, real world practical feedback and a bunch of reviews from photographers who have used the gear is useful. Scores, not so much.

Hmm. Those guys didn't shoot birds, animals, fashion, macro, print large detailed landscapes. For those that do, DXO Mark is valuable.

[martycon]

Rick, I thank you and all others, for your informative and respected comments. Yes most decisions are a compromise, and folk who see only in B&W probably get more sleep.
Many of you use DxOMark and the like as a tool in decision making, and I am now comfortable to do likewise. As Arthur comments, AP members are a great repository of knowledge photographic.

[Hamster]

Hmm. Those guys didn't shoot birds, animals, fashion, macro, print large detailed landscapes. For those that do, DXO Mark is valuable.

Sure they did ( not those two specifically :-)) but it was all still done. And the best did it with gear that wouldn't even reach the bottom of the scale now. The gear they used wasn't what made them icons, and it won't be what does they same for anyone reading a dxomark score.
Note, I'm agreeing with the consensus here, just saying that in the grand scheme of things I believe a score to be pretty insignificant. A full frame sensor and L glass has sure made my life easier on the technical side, but capture the imagination of your audience and they don't care one iota if there's a bit of vignetting out wide. In fact, it's ironic that with today's fantastic optics people will add vignetting or an out of date film effect. Again, I'm not saying I don't like them, just observing the irony.

[Nick Cliff]

I have to say I agree that "wettems" Rockwell is a good laugh and a very rough guide from my experience. Two sites I have found interesting and perhaps less biased if not mentioned previously are the LensRentals site in the (USA) and Lens Tip in (Poland). Both sites also have historical knowledge that may be of some interest.

[arthurking83]

.... Two sites I have found interesting and perhaps less biased if not mentioned previously are the LensRentals site in the (USA) and Lens Tip in (Poland)..... .

Agreed!

Lens Rentals are great(mainly for lenses, but some other stuff too) because of the overall and general breadth of the reporting.
ie. large sample collection that averages out the good and the bad : reporting of what you can expect at the top end with the good, and what to look for at the bottom end with the bad.

AFAIK tho they don't really do camera comparisons as such .. more so durability and technical issues.

I've seen Lens Tip, and remember they did some interesting comparisons with some filters which seemed to be good info.
I haven't really read many of their other camera/lens reviews tho.

[Lance B]

Sure they did ( not those two specifically :-)) but it was all still done. And the best did it with gear that wouldn't even reach the bottom of the scale now. The gear they used wasn't what made them icons, and it won't be what does they same for anyone reading a dxomark score.

And most of it was pretty terrible compared to todays high quality images. The stuff that we see now from the modern cameras of birds, animals, fashion, landscapes etc would blast the previous images that you talk about into the weeds. The point is not so much that you may be able to get decent images with yesteryears cameras, just that it can be done so much easier and most importantly, it performs when you want it to and will rarely if ever let you down. That cannot be said for the old film style cameras. There is no doubting this fact.



Note, I'm agreeing with the consensus here, just saying that in the grand scheme of things I believe a score to be pretty insignificant. A full frame sensor and L glass has sure made my life easier on the technical side, but capture the imagination of your audience and they don't care one iota if there's a bit of vignetting out wide. In fact, it's ironic that with today's fantastic optics people will add vignetting or an out of date film effect. Again, I'm not saying I don't like them, just observing the irony.

I think you're mixing up two basically different styles to photography. There are those photos that do not need high quality technical results to make them work, mainly just relying on light, composition and subject matter/story. Henri Cartier Bresson would not care much about the camera he uses but he wouldn't need to with the photos he takes and the effect he wants to portray. However, there are those photos that most certainly do require high quality technical aspects to them, where resolution, colour accuracy, high ISO noise quality etc is of paramount importance, like I mentioned, birding, animals, fashion, landscapes etc. This is where the current crop of cameras excel. This cannot be disputed and this is what DXO Mark et al are about. The point is, if you have the hi res super fast focusing camera of today, it is easier to make the photo retro look by adding vignetting and adding noise or going B&W than it is by using a manually focusing camera and trying to sharpen up a blurry shot or removing the noise, or changing ISO mid-shoot, or whatever.

[Hamster]

...There is no doubting this fact. ....This cannot be disputed....

I'm afraid you appear to be responding to far less subtle points than the ones I was actually making. But that's OK because, despite you helpfully pointing out the bits I shouldn't bother disagreeing with, I can see that we have (subtly) differing opinions on the subject. And isn't that great because life would be so boring if we all had the same opinions. If it helps, I can tell you that I agree with the bloke who wrote post #5 :)

[Arg]

When considering purchasing lens or body, do you use DxOMARK or similar objective measurement publications to help you arrive at a decision? Are there others that you can recommend.
I am impressed with DxOMARK, but am not sure whether my trust is misplaced.

If you always buy the models with the latest, gushiest reviews and the best measurements on internet test sites, at least there will be some interest when it's time to sell on ebay (when an even better model with even better measurements is released). :p

If you don't care about resale, however, then listen to no-one. Your needs and sensitivities are yours alone. You might be driven mad by a certain handgrip -- but your friend recommends the camera because he has numb meataxes for hands and would only comment if the camera actually fell from his grip. Who cares if the sensor would have allowed you to shoot at 1/2 stop slower ISO, if you kinda like the look of ISO6000 and never go that high anyway? Who cares if the corners are slightly softer at f/2, when you are happy to use it at f/5.6 where it is pin sharp? Know what photos you want to make, then shop by feature set that will deliver your photos, to a budget.

Within the limits of 'get what you pay for', they all deliver as per the feature set. Anyway, if you are into a camera system, it's no joke to swap systems. Then what: the company you just sold out of releases a new model that beats on score the model (and system) you just bought into (at great cost)? Now what?

As for DxOMark specifically, no I don't like it. Sometimes their numbers are wrong - rarely, but how do you know whether the specific number you are looking at isn't one of the few wrong ones? And when right, they are very misleading at the aggregate levels. More fights online over DxOMark scores than any other site. That tells me that they could have done it better -- much better.