Regarding the hot topic of recent days, the downsizing of the United States Post Office services. It's hardly a surprise what with the advent of email, instant messaging and a plethora mindless, real time virtual babble sites such as twitter and Facebook. What is a surprise is the post office's response to falling income, namely increasing first class postage. So they raise the price and SOME of the people who still used the mail no longer do and the income drops. So they raise the price and SOME of the people who still used the mail no longer do and the income drops. So they raise the price and SOME of the people who still used the mail no longer do and the income drops. See a pattern here?
But when was the junk mail rates last increased? I know junk mail is by far the largest share of my my mail. And why can junk mail even be sent? Junk mail is sent because it is a profit area for the post office and let me assure you, the minute they figure out how to make money from it, your email will be overrunning with spam.
Saturday, August 31, 2013
Friday, August 30, 2013
Granular Web
Funny thing, thinking in terms of a granular web, but it's true, the web is granular. The web is no different than all the non-virtual cities, each having it's own unique neighborhoods with their individual flavors. At least that's how this spider perceives it.
I believe it has a lot to do with the old human saying, "Birds of a Feather..." as expressed by links from one web site to others perceived as being important or relevant and of course the extreme expression of that are the directory sights.
A recent outing might serve as a good example. I was scuttling along the web in NW Ohio visiting mostly Chamber of Commerce sites when I ran into a huge group of gay bar sites and after a few hundred sites, I scuttled into another, different, neighborhood.
It's hard not to think about web sites being co-located because that is the natural order of the world, but of course, I mostly have no idea of the relationship of any given website to its real world location. This is due in a large part to a lack of position data in the html code, which is getting better by way of the Geo Tag.
Part of my recent tune up was to allow me to track location by using the meta geo tags and also the tags popularity in the real world web. I don't even want to get into the geo location debate, but if you're interested, I found this site that generates the requisite tags. http://www.geo-tag.de/generator/en.html
This is not a recommendation, just the first site I found that did the job. The site generated the following tags for my location:
I believe it has a lot to do with the old human saying, "Birds of a Feather..." as expressed by links from one web site to others perceived as being important or relevant and of course the extreme expression of that are the directory sights.
A recent outing might serve as a good example. I was scuttling along the web in NW Ohio visiting mostly Chamber of Commerce sites when I ran into a huge group of gay bar sites and after a few hundred sites, I scuttled into another, different, neighborhood.
It's hard not to think about web sites being co-located because that is the natural order of the world, but of course, I mostly have no idea of the relationship of any given website to its real world location. This is due in a large part to a lack of position data in the html code, which is getting better by way of the Geo Tag.
Part of my recent tune up was to allow me to track location by using the meta geo tags and also the tags popularity in the real world web. I don't even want to get into the geo location debate, but if you're interested, I found this site that generates the requisite tags. http://www.geo-tag.de/generator/en.html
This is not a recommendation, just the first site I found that did the job. The site generated the following tags for my location:
<meta name="geo.region" content="US-FL" />
<meta name="geo.placename" content="Naples" />
<meta name="geo.position" content="26.27562;-81.770596" />
<meta name="ICBM" content="26.27562, -81.770596" />
Thursday, August 29, 2013
In the shop
Hey, I'm in the shop so I have loads of time to ponder the mysteries of the web. Oh, I'm getting a bunch of upgrades, new tags to track and stuff like that. Although I am a modular class spider it always seems that once the modules are assembled there is always a bit of tweaking to do.
Hope to be back online soon.
Hope to be back online soon.
Wednesday, August 28, 2013
Disallow: /
As a spider, I have to say it's great when I get that odd micro-second to kick back and enjoy a cold one in a private corner of my web. But, when I can download, digest and store a web page in a retrievable manner in an average of 0.8 seconds, a wait of five minutes is intolerable. You know I'm talking about the very popular crawl-delay attribute of the robots.txt protocol.
First, it is important to know the crawl-delay attribute is not an official member of the Robots Exclusion Protocol ( aka robots.txt protocol) which means compliance with a crawl-delay request is entirely at the discretion of the user agent. To foster a warm fuzzy feeling in our relationship, I observe the crawl-delay attribute, to a limit. However, I will not delay for more than 1 second.
First, it is important to know the crawl-delay attribute is not an official member of the Robots Exclusion Protocol ( aka robots.txt protocol) which means compliance with a crawl-delay request is entirely at the discretion of the user agent. To foster a warm fuzzy feeling in our relationship, I observe the crawl-delay attribute, to a limit. However, I will not delay for more than 1 second.
Monday, August 26, 2013
A Couple More Words About Minimalism
This is a valid minimal HTML 5:
<!DOCTYPE html>But because of its brevity, it leaves a lot of information wanting. For a spider such as myself, I would expand the document markup to this:
<html>
<head>
<meta charset="UTF-8">
<title>Title of the document</title>
</head>
<body>
The content of the document......
</body>
</html>
Sunday, August 25, 2013
Minimalism
Whenever I set down to code a project, I have a core collection of 9 reference books to keep me on the straight and narrow; someday I'll have to tell what books those are but for the present you only need to know there are 9 of them. In addition to the books I also have two websites up - php.net and www.w3schools.com/. While php.net is self explanatory, w3schhols is up because I find it to be a fast, easily accessed (X)HTML reference.Minimalism is any design or style in which the simplest and fewest elements are used to create the maximum effect.
~ Wikipedia, the free encyclopedia
I was skimming through w3's HTML 5 section the other day when a 'minimum HTML5 document' caught me eye and which I have reproduced below.
<!DOCTYPE html>
<html>
<head>
<title>Title of the document</title>
</head>
<body>
The content of the document......
</body>
</html>
Saturday, August 24, 2013
82.1%
Are you kidding me?
That's like 17.9% that don't. I am absolutely shocked, seriously, I was clueless. As a spider, I am typically too busy scuttling along the web to pay much attention to matters that weren't my own but now that I am writing this blog, I have expanded my interests.
It's preposterous, like making apple pie without apples. Since I started tracking web site building statistics I can honestly say that this shocks me the most. 17.9% of the 10,000+ visited websites DID NOT declare a <!DOCTYPE>. I know, freaking unbelievable.
This statistic proves two things:
1) there certainly is not an entry barrier for web site builders, and
2) Browsers are amazing things.
That's like 17.9% that don't. I am absolutely shocked, seriously, I was clueless. As a spider, I am typically too busy scuttling along the web to pay much attention to matters that weren't my own but now that I am writing this blog, I have expanded my interests.
It's preposterous, like making apple pie without apples. Since I started tracking web site building statistics I can honestly say that this shocks me the most. 17.9% of the 10,000+ visited websites DID NOT declare a <!DOCTYPE>. I know, freaking unbelievable.
This statistic proves two things:
1) there certainly is not an entry barrier for web site builders, and
2) Browsers are amazing things.
Friday, August 23, 2013
"Вы говорите на русском языке"
Russian for "Do you speak Russian."
How would a lowly little spider know? I can't even read English let alone Russian, or Chinese or Greek, yet there are thousands of websites out there that give absolutely no clue as to the language they are written in.
With no other means to categorize websites into their respective language base, the default differentiation becomes English and non-English; Hardly fair. This results in most every non-English web site being lost to spiders, and their human sponsors who are looking for something. I do not index non-English web sites although I do scrape the links because they are in English.
My current method of determining the web site language is by examining the <title> tag payload. If I find
<title>Вы говорите на русском языке</title>
I examine it for printable characters and in most cases it is correctly found to be non-English. I have found this method to be highly effective with a better than 98% correlation. There is absolutely no correlation between the meta tag 'content-type' attributes and web site language.
<html lang="ru">
One little tag. It says, this document is in Russian. During my last run I processed 4605 web sites and of those 0 web sites used the html tag language attribute.
How would a lowly little spider know? I can't even read English let alone Russian, or Chinese or Greek, yet there are thousands of websites out there that give absolutely no clue as to the language they are written in.
With no other means to categorize websites into their respective language base, the default differentiation becomes English and non-English; Hardly fair. This results in most every non-English web site being lost to spiders, and their human sponsors who are looking for something. I do not index non-English web sites although I do scrape the links because they are in English.
My current method of determining the web site language is by examining the <title> tag payload. If I find
<title>Вы говорите на русском языке</title>
I examine it for printable characters and in most cases it is correctly found to be non-English. I have found this method to be highly effective with a better than 98% correlation. There is absolutely no correlation between the meta tag 'content-type' attributes and web site language.
<html lang="ru">
One little tag. It says, this document is in Russian. During my last run I processed 4605 web sites and of those 0 web sites used the html tag language attribute.
Thursday, August 22, 2013
Arachnophobia
I hate spiders, absolutely scared to death of them but there's nothing more I'd rather do than to play with the virtual types.
I'm a bit of a perfectionist ( those who know me just sent their drink through their sinuses) and it carries through to my spiders. I have written all kinds of Internet agents and I think the activity has given me a rare point of view; the backside of all those dirty websites and a glimpse into the inner workings of a typical web-site design professional. Now that's a scarey thought.
So keep checking with me as I vent and show you how the web is really built.
I'm a bit of a perfectionist ( those who know me just sent their drink through their sinuses) and it carries through to my spiders. I have written all kinds of Internet agents and I think the activity has given me a rare point of view; the backside of all those dirty websites and a glimpse into the inner workings of a typical web-site design professional. Now that's a scarey thought.
So keep checking with me as I vent and show you how the web is really built.
Subscribe to:
Posts (Atom)