Two friends of mine are about to buy a domiciliary care business, and over dinner the other week we discussed their website and how effective search engine optimisation can ensure it shows high in any Google search.

Before long we were lamenting Google’s business practices and commercial dominance, something I blogged about in late 2011. I had lots to say about this. Earlier that week I’d returned to work after a week’s holiday and learned that Google had de-indexed bmj.com, apparently without notice.

Google had accused us of “cloaking.” This means we were allowing Google’s search engine spiders to crawl our full text content, but because all of our non-research content is now behind a paywall (a change we made in January 2012), a typical end user landing on our content after clicking on a Google link would only see a 150 word extract.

We’d got round this by delivering a longer pdf extract. This was one of Google Scholar’s recommendations when we met with them in December, and we’d understood that this would keep google.com happy.

Only it hadn’t kept them happy. This was serious stuff. Before long authors and readers were emailing us to ask why BMJ articles weren’t appearing in Google search results. There was an immediate impact on our traffic. In one week in February it dropped from 335 942 to 239 918 visits. A week later we lost another 10 000 visits.

According to Google Scholar, the pdf extract option should only have applied to scholarly research articles lacking abstracts, and not to non-scholarly content, such as book reviews, letters, editorials and general news articles. For non-scholarly content, we could have introduced Google’s first click free option. This means that if someone clicks on Google search result they hit the article rather than a paywall, but if they then browse within the site the paywall launches.

Before being de-indexed, bmj.com had enjoyed a good page ranking for its content. This means we showed high in Google’s search results. We’d heard it can take years to regain this ranking. But first we needed to get re-indexed urgently.

We decided to let Google index free content (mostly original research) and abstracts/extracts of everything else. We applied this fix urgently, and alerted Google.

The good news is we’re now being reindexed, and according to Google Scholar, our high page ranking should return quite quickly because our removal period was relatively short (three weeks).

When Google Scholar representative Darcy Dapra addressed a meeting of scholarly publishers last year, its indexing rules led to accusations that the company was dictating to publishers how they should run their businesses. The company argued that publishers weren’t being singled out – its indexing rules were applied across the board.

Ultimately it’s for businesses to decide how important Google traffic is to them. If it isn’t, don’t let Google index your site.

Where does this leave my friends setting up their business? They want their website to be search engine optimised, but it rankles with them that Google calls the shots. We decided over dinner to break the company’s stranglehold by using an alternative search engine (Bing, Yahoo etc), at least once a day.

Have they done that? I doubt it. Have I? No, not yet. Will I? Perhaps not. If I were truly serious I’d close my gmail and Google+ accounts. I’d suggest that my colleague emails me the joint document we’re working on, instead of sharing it via Google documents. And I’d stop my fixation with bmj.com’s Google Analytics account to check our web traffic most days.

Google is a hard habit to break.

David Payne is editor, bmj.com