Russian for "Do you speak Russian."
How would a lowly little spider know? I can't even read English let alone Russian, or Chinese or Greek, yet there are thousands of websites out there that give absolutely no clue as to the language they are written in.
With no other means to categorize websites into their respective language base, the default differentiation becomes English and non-English; Hardly fair. This results in most every non-English web site being lost to spiders, and their human sponsors who are looking for something. I do not index non-English web sites although I do scrape the links because they are in English.
My current method of determining the web site language is by examining the <title> tag payload. If I find
<title>Вы говорите на русском языке</title>
I examine it for printable characters and in most cases it is correctly found to be non-English. I have found this method to be highly effective with a better than 98% correlation. There is absolutely no correlation between the meta tag 'content-type' attributes and web site language.
<html lang="ru">
One little tag. It says, this document is in Russian. During my last run I processed 4605 web sites and of those 0 web sites used the html tag language attribute.
No comments:
Post a Comment