SEO seems, on the face of it, like a numbers game. How many links do you have? How many words of content have you produced? How many videos do you have on YouTube? But there’s a lot more to SEO — both on- and off-page — that is more qualitative.
For example, if you’ve got twice as many backlinks as the guy listed just above you, you should be taking a look at the listings themselves. Maybe yours says “Bird’s Eye Peppers Thai Peppers Dried”, and his says “5 Awesome Dishes that Use Thai Bird’s Eye Peppers”. You clearly have more rankable keywords in your title, but remember that Google tracks clicks as well as keywords — if his title is getting him the clicks, it’ll eventually drive his rank as well.
If you over-optimize your content for SEO purposes, you’re shooting yourself in the foot. The key is to create content that is usable, awesome, and still SEOed enough that the search engines will notice that it’s awesome.
Similarly, your brand has a noticeable impact on your ranking as well. If your brand just isn’t getting any recognition (i.e. no one searches for it and then clicks on it), it’s just another piece of useless content. Tests have proves that searches who recognize a given brand are much more likely to click on it — and again, with Google counting clicks, having a strong brand will eventually raise your rankings just by being a strong brand.
Technically, this may be quantitative information, but it’s at least partly qualitative as well — social media mentions, be they URL mentions, brand mentions, product mentions, or other associated mentions, are tracked by Google as well. If you have a scandal and the social media sphere erupts with anti-you sentiment, you can kiss your ranking goodbye — similarly, if a competitor clearly has less banklinks and poorer SEO but is still outranking you, their social media campaign may be the culprit.
One of the worst possible signs that you can show Google is having someone search you up, find your page, and then ‘bounce’ back to Google (usually via the ‘back’ button) and immediately choose a different link. That tells Google that your page either didn’t have the relevant material they wanted, or that your web design was so bad that they couldn’t find what they wanted. Now if your site really just isn’t appropriate for that search, that’s one thing — but if your web design is holding your rankings down, that’s a huge issue, and one that looking at the straight numbers may never reveal.
The short version of it is, if you’ve got a website and it looks like it’s SEOed properly and has more backlinks than it’s competitors — and you’re still not outranking your competitors — there’s probably a quality issue somewhere in your Web presence. Start at the top and work your way down, and see if you can pick out where you can qualitatively improve what you’re doing.