There are a lot of spelling mistakes that annoy me, in addition to your and you're. Words such as definitely, unfortunately and the uses of been/being, and were, where and we're. I read those on message boards all the time, particularly sport forums.
It seemed from your post (perhaps I mis-interpreted it) that all characters in the series should speak perfect English unless they are a copper? Or do you perhaps mean that at least some of the characters should be a bit more linguistically savvy?
Other than dialogue I agree the grammar should be more accurate, if not for my sake (because I can't remember noticing
) then perhaps for other people who do.
Language evolves too and phonetics play a big part in that and the simple reason for the 've's and 'of' is because they're close enough in pronunciation to account for the laxness and/or confusion ('uv' and 'ov' not being a million miles apart in some regional British dialects). 
) and mark the journey of language across the globe with words like 'plough' morphing (or perhaps remaining even static?) in spelling as 'plow'... across the Atlantic. That's all 'of' and 've' is in essence and so long as people know what it means it AIN'T doing too much harm is it? 
Blimey - we agree! 