ZOMDir > Blog

Saturday, 14 October 2017

Dead Link City - A comparison of 8 Free Online Link Checkers

The site DeadLinkCity.com is a test site used by DeadLinkChecker.com.

I used this site to improve the link checker Broken Links at a Glance. Now it as good as DeadLinkChecker or, in my humble opinion, even better.

Take a look at these results.

1. Broken Links at a Glance
Tests 86 unique links, finds 75 broken links

2. DeadLinkChecker
Tests 95 links, finds 74 broken links

The difference between these link checkers is a link to the page http://www.deadlinkcity.com/disallowed/disallowed.html

This page should be blocked by robots.txt but isn't due to malformed code.

Instead of:
User-agent: *
Disallow: /disallowed/disallowed.html

this robots.txt file contains the code:
User-agent: *
Disallow: disallowed/disallowed.html

The difference of one "/" is the difference between blocked or not. See for yourself what a difference a "/" makes with this Robots.txt Testing Tool.

You might also use this Robots.txt Test Tool to see live if this "disallowed" page is really disallowed.

So DeadLinkChecker interprets the robots.txt file of DeadLinkCity incorrect, at the moment of writing this blogpost.

3. Online Domain Tools
Tests 85 links, finds 68 broken links

This link checker misses the following broken links:
  • disallowed/disallowed.html
  • error-page.asp?e=401
  • images/missing-command-icon.jpg
  • images/missing-video-poster.jpg
  • missing-button-formaction.asp
  • missing-head-profile.txt
  • missing-html-manifest.txt

4. W3C Link checker
Tests 79 links, finds 68 broken links

This link checker misses partly the same broken links. The missed broken links are:
  • disallowed/disallowed.html
  • images/missing-input-src.jpg
  • missing-button-formaction.asp
  • missing-form-action.asp
  • missing-input-formaction.html
  • missing-object-classid.html 
  • missing-object-codebase.html 

5. Dr. LinkCheck
Tests 51 links, finds 46 broken links

6. BrokenLinkCheck
Tests 49 links, finds 43 broken links

7. Internet Marketing Ninjas
Tests 50 links, finds 39 broken links

8. WebToolHub
Tests 54 links, finds 9 broken links

Good luck with your links, choose your link checker tool wisely and don't forget to check your links at a regular base.

Hans

--
ZOMDir.com is a dynamic directory and a wiki
Everyone is able to add a link in 10 seconds
To learn more view this Slideshare presentation

Thursday, 12 October 2017

The half-life of a link is two year

The half-life of a link is two year. Better said, the half-life of an external link is two year. 

That is, when you create today a website with 100 working external links and checks your website after two year with a broken link checker, you will discover that rougly 50 links are broken.


How do you know?

I can almost hear you thinking "How do you know?". Well I will explain below.

In the past I have copied as much data as possible of the directory Yahoo! This is because Yahoo! stopped, I have created a directory myself and I wanted to analyse the links and structure of this famous directory.

At January 4, 2016 I analysed the data I have and concluded that 77% (or more exactly 76.8387682%) of the links are fine.

Recently (October, 9 2017) I analysed the data again. Now 42% (42.0219319%) of the links are fine.

Based on this data I concluded that on an average day 0,093670021% of external links will get broken. That does not seem much. However the linkrot percentage per month is 2.81%. 


Consequences

After a half year one sixth of the links are broken.
After a year 30% of the links are broken.
After two years 50% of the links are broken. Hence the half-life of a link is two year.

See also this graph below



So when you think 3% broken links is acceptable, then you should check for broken links every month.

When 5% is acceptable, check every two months and when you think 10% is acceptable, check every 4 months for broken links.

Be wise, and check and repair your links at a regular base,
Hans

Update: After writing this blogpost  I discovered that in the document "A longitudinal study of Web pages continued: a consideration of document persistence" it is stated that the half-time of a random web page is about 2.0 years. Great that's exactly what I concluded.  

--
ZOMDir.com is a dynamic directory and a wiki
Everyone is able to add a link in 10 seconds

To learn more view this Slideshare presentation

Monday, 9 October 2017

How DeadLinkCity improved "Broken Links at a Glance"

DeadlinkCity is a test site of www.deadlinkchecker.com.

The site contains several types of broken links similar to httpstat.us/

At DeadLinkCity it is stated that:
(...) There are 74 known bad links in DeadLinkCity.com, and one additional link which should not be reported if the tool obeys robots.txt directives. (...) The perfect score is 74 - the closer the number of reported errors is to 74, the more accurate the tool is. (...)

When I tested DeadlinkCity with "Broken Links at a Glance", the link checker discovered 56 broken links. That is far from perfect. 

Quickly I discovered that "Broken Links at a Glanceonly checks links mentioned in the href and src attribute.

So I decided to modify "Broken Links at a Glance". Now it also checks links which might be mentioned in the attributes:
  • action
  • archive
  • background
  • cite
  • classid
  • codebase
  • data
  • formaction
  • icon
  • longdesc
  • manifest
  • poster
  • profile
  • srcset
  • usemap

This list is based on these overviews of HTML 4 attributes and HTML 5 attributes.

Retesting DeadlinkCity with "Broken Links at a Glance" gives the almost perfect score of 72 broken links found.

Nice but it seems that I still miss the 3 URL's mentioned in the CSS code. So I updated "Broken Links at a Glance" that it also checks for links in the CSS code. The tool now finds 75 broken links.

That's one to much.

According to DeadLinkCity disallowed/disallowed.html shouldn't be tested, because it was mentioned in their robots.txt file. 

However the robots tester I use is very strict. It does allow this link because it didn't start with a "/".

Hopefully "Broken Links at a Glance" will be added to the Comparison Table of DeadLinkCity soon (and their robots.txt file will be corrected).

Hans

--
ZOMDir.com is a dynamic directory and a wiki
Everyone is able to add a link in 10 seconds

To learn more view this Slideshare presentation