The inner workings of a search engine are a corporate secret. The general purpose of search engines and how they work, are not.
In today’s post, I will provide a non-technical overview of search engines. I will discuss what search engines do, how they do it, and why they do it. I will also discuss when and where.
You may watch the Periscope broadcast of this post on YouTube.
What Search Engines Do
Search engines like Google and Bing index websites and other content in order to make those websites and digital objects findable. They do not store the website itself. The systems do not store the digital objects themselves.
They simply store how to find that website and/or digital objects. Yes, like an address book, but they have connections — links — that will take you to the location of the website and/or digital objects.
How Search Engines Index the Web
They search the web by using something called crawlers or spiders. These “robots” traverse the web, going from site to site by way of links between websites.
As the crawlers discover new pages or find that old pages are no longer there, this information is passed back to the index. While the crawlers are going through each site, they are constantly making assessments about the quality of the site structure and content.
Crawlers also examine how often a site is updated. Contrary to what seems logical, the more often a page is updated, the less often it is crawled.
The freshness factor, as this is called, may be high, but from a technical perspective, it does not make sense to constantly crawl a site that is constantly changing. What the crawlers do, therefore, is visit the pages on a regular basis, and take snapshots of the content at that time.
The questions the crawlers step through involve what keywords are on the page. The spiders look at whether or not those keywords are in the title, the URL or directly adjacent to links or highly used keywords.
Each crawler examines how well the site is structured. It takes into account things like load speed and code quality. Crawlers also examine citation analysis.
In the Google World, this is referred to as PageRank. This is the “why” factor. This is the reason crawlers do what they do. Citation analysis is so important, I’ll devote the next section to it.
In a previous post, I used the example of the sections of a paper to describe how to apply keywords to website design.
For this section, I’ll talk about the part most of us disliked — adding in the citations to a paper. Here’s an example from a paper I wrote a few years ago.
Now, what do these boring old things have to do with search engines?
It’s simple, actually. Let’s say you are in a class of 30 people. You each have to write a 20-page research paper for the course. Fifteen students cite the same three authors in their papers.
If the teacher analyzes the citations for the papers, he or she will see similarities in the references. Does this mean the students cheated? Well, we hope not.
What it should mean is that these three authors the fifteen students cited are considered authorities. They are important. The work of these authors matters.
Now, there are many ways to look at this. “Everyone” could be talking about those three authors because their work is bad. The papers are actually critiques. Or, the students could be citing the same authors because their work is excellent.
It is up to the teacher to decide why the students cited these authors based on the context of the paper. If the cited authors are controversial, it could be that 1/2 the students citing the authors are “for” the authors, and the other 1/2 are critical. It depends.
The point, however, is that these authors are cited because their work is important for some reason.
Citation analysis is the point of Google’s PageRank algorithm. (My explanation is an oversimplification, but it gets the gist of it.) PageRank is the Holy Grail for people who want their pages to rank well organically.
What You Can Do to Rank Well
You see citation analysis every day. Do you have friends and/or relatives to whom you go for advice? Are there other people whom you avoid like the plague because their advice is bad? Or you know some friends and relatives are somewhere in between?
Yes, that is citation analysis. Yes, that is PageRank.
If you want to make an A in the class, attend the class. Pay attention. Talk to the teacher/professor. Learn something. Synthesis it. Actually, do the work. Make an effort. At worst, you’ll make a B.
The same goes for your website. If you want to rank well, it is pretty simple.
Have a well-designed site with solid content that is technically sound.
Use the appropriate keywords as the basis for your content. Link to sites with good content. Build your authority so that you will be linked to by other quality websites with good content.
By the time you go to the effort of buying links or keyword stuffing and applying dirty SEO methods, you could have taken the time to create a site that visitors actually want to see with content they want to read or watch.
Optimizing your site for search engines to find and rank well really is that simple. You want the site to be technically sound. You want users to be able to navigate it well. Your users want interesting content that is simple to read and engaging. That’s it.
Of course, the simpler something is, the harder it is to do well and the more skill it takes to do. However, as Nike says, “Just Do It“.
If you would like to get found online, work with me on your small business SEO.