The Better Way to Hide Links from SEO Crawlers

It’s no question that hiding links is critical to any successful SEO campaign.

When building links, especially on PBNs and the like, it’s important to block SEO crawlers like Ahrefs (ahrefsbot) and Majestic (MJ12Bot). Most SEOs accomplish through abhorrent .htaccess lists of bots to block.

what are all of these bots anyway?

The current solutions are…

  • .htaccess files
    • Not always applicable if you don’t have full control over the site
    • Often times blocked from being edited
    • Require some technical know-how, updating, and blind trust
    • Footprint if set up incorrectly
      • Blind trust / copying pasting code you find online is bound to lead to problems
      • Most .htaccess codes available online redirect bots to webMD.com
  • robots.txt
    • (lol)
  • WordPress plugins
    • Not so great if you’re not using WordPress
    • Require blind-trust in the author of the plugin
    • Footprint if (when) set up incorrectly (more on that in another post)

Using Javascript, we can abuse the achilles heel of crawlers and have links “hidden” in the source code of a site, but visible on the page load.

This is what the SEO crawlers will see (no visible links), but here is what Google will see.

Because we broke the link up into two parts (variable a and variable b) and combined the variables when the page loaded, we were able to hide our link from third party SEO crawlers.

This requires little to no technical knowledge and relies on simple Javascript. No sneaky redirects or cloaking.

This is helpful for those who are also building links on public facing properties and can place Javascript: most website builders and submission platforms allow Javascript embed within the HTML, but not access to .htaccess or robots.txt. Using this, you can block your competitors from viewing more than just your PBNs!

You can view the code for this here. Replace link with your anchor text, and replace the URL split among the two variables with your own.

After doing so, simply invoke your link by using <span id=“target”></spaninstead of the hyperlink HTML code.

Bulk Twitter Username Availability Checker

This is a free tool that uses Twitter’s undocumented username availability checking API to verify the availability of a large list of Twitter usernames.

To use, create two files: input.txt and proxy.txt. The former should contain a list of usernames, one per line, that you want to check, while the latter should contain a list of proxies, one per line. Actions are logged to the console (visible window so you can see what is going on) and the list of available usernames are saved to output.txt.

The program is designed to check one username per proxy every five seconds: safe limits to keep the program running without worrying about limitations. Although not recommended, because the source is included, this number can quickly be modified by those with basic programming knowledge.

A visual of the program in action is available here (refresh a few times if it doesn’t load), while the program can be downloaded here. For this program to function properly, need the latest version of the Microsoft .NET framework.

Those who act quickly can take advantage of this free data from Majestic (reuploaded, Majestic 404) to find juicy Twitter accounts that are available for registration. This list is from 2012 and sorted by Majestic metrics, meaning the users in the list have had enough time to change their usernames, leaving powerful usernames available to register.

This tool was created because using software like Gscraper and Scrapebox is inefficient and inaccurate. The abovementioned softwares request a whole page to get the status code, and often times are timed out and return an empty (–) response, leaving you with no idea if the username is available or not. Additionally, using those softwares to check for 404s returns inaccurate results, as usernames that return as 404s are not necessarily ones that are available to register (such as banned accounts, reserved usernames [BritishMonarchy, for example], and URLs that don’t exist, but are not usernames available to register).

This tool is light years faster and not even comparable in terms of accuracy to the above mentioned solutions.

Oh, and it’s free. 😉