Monday, March 28, 2005

An interesting useless scraper site.

I've noticed this site: ht*p://www.traffic2000.com.cn . (There are also other identical sites on other domains) It's basically a series of pages (Many thousands of them), with each page directed at a certain random popular keyphrases. The content of the page is: A series of meaningless pictures of fish and cars, the keyphrase is repeated a couple of times, with adwords ads, and a list of links and descriptions for top ranking sites for that particular keyphrase. These look to have been scraped from search results.

Pages from this site are coming up pretty high in the search results for some pretty big terms, and I got curious as to how such a crap site could outrank so many legitimate sites. I did a backlink check in Google, and it shows 1460 backlinks, all from what are legitimate sites which, after inspecting the source, do not link to traffic2000.com.cn.

This guy is somehow successfully convincing G that other sites are linking to him when in fact they are not. I'm sure it won't be long before his site vanishes from the index, I've noticed the ads have been disabled and are just showing comunity service ads, so it can't be making much money anymore.

Saturday, March 19, 2005

Content

I'm just packing my pages with thousands of pages of good content that I just acquired. There's nothing better than unique text content that you don't have to type out yourself when it comes to getting people to a site. Content is what web design is all about. When I go to the internet, I go to find something out. I don't really care about the themed backgrounds and banners, buttons and textures that take ages to load on a modem. (Yep, I still use one at home.... but not for long hopefully.) As long as you know what the page is, it looks good, and loads reasonably quickly I'm happy. Looking foreward to all these pages hopefully getting crawled and indexed, and seeing if anyone is interested in them.
I'm bloody starving. I'm going to Hungry Jack's:) It's Sunday.

* I Don't get paid or anything to promote hungry jack's or anything, there's just one up the road.

Friday, March 11, 2005

302 redirects and Googlejacking

There are currently a lot of angry webmasters up in arms about the way G is handling 302 redirects.

The effect of this problem is that one can damage another site's rankings in Google simply by "linking to it in a special way" ie; redirecting to it using a code 302 redirect. to put it as simply as possible.

It seems that it is almost as easy as that. Many sites use 302s to link out to other sites for various legitimate reasons, even Yahoo and Alexa use such a linking scheme. For some reason G seems to know that these big players are linking out and doesn't create a new index with the content of the target page, and the URL of the redirect script, but on some smaller directories I have noticed this is happening. Im still not sure if it's something in the way the 302 is executed that is causing the double indexing, or if Google just knows Alexa and Yahoo. I have found a couple of directory sites that I reciprocate links with that are coming up under a search for my content. I have asked them to cancel the links, of course. I recommend only exchanging links with sites that use proper "a href" hard links.

Here is a great post from a Webmaster World forum thread which explains the details of the problem much more concisely than I am able:


The full story of Google and 302s


You can't ban 302 referrers as such


Why? Because your server will never know that a 302 is used for reaching it. This information is never passed to your server, so you can't instruct your server to react to it.

You can't ban a "go.php?someurl" redirect script


Why? Because your server will never know that a "go.php?someurl" redirect script is used for reaching it. This information is never passed to your server, so you can't instruct your server to react to it.

Even if you could, it would have no effect with Google


Why? Because Googlebot does not carry a referrer with it when it spiders, so you don't know where it's been before it visited you. As already mentioned, Googlebot could have seen a link to your page a lot of places, so it can't "just pick one". Visits by Googlebot have no referrers, so you can't tell Googlebot that one link that points to your site is good while another is bad.

You CAN ban clickthrough from the page holding the 302 script - but it's no good


Yes you can - but this will only hit legitimate traffic, meaning that surfers clicking from the redirect URL will not be able to view your page. It also means that you will have to maintain an ever-increasing list of individual pages linking to your site.

For Googlebot (and any other SE spider) those links will still work, as they pass on no referrer.

 











This is what really happens when Gbot meets 302:


Here's the full lowdown. First time i post it all. It's extremely simplified to benefit the non-tech readers among us, and hence not 100% accurate in the finer details, but even though i really have tried to keep it simple you may want to read it twice:

 



  1. Googlebot visits a page holding eg. a redirect script

  2. Googlebot indexes the content and makes a note of the links

  3. Links are sent to a database for storage until another Googlebot is ready to spider them. At this point the connection breaks between your site and the site with the redirect script, so you (as webmaster) can do nothing about the following:

  4. Some other Googlebot tries one of these links

  5. It receives a "302 Found" status code and goes "yummy, here's a nice new page for me"

  6. It then receives a "Location: www.your-domain.tld" header and hurries to that address to get the content for the new page.

  7. It deliberately chooses to keep the redirect URL, as the redirect script has just told it that the new location (That is: your URL) is just a temporary location for the content. That's what 302 means: Temporary location for content.

  8. It heads straight to your page without telling your server on what page it found the link it used to get there (as, obviously, it doesn't know - another Googlebot fetched it)

  9. It has the URL (which is the link it was given, not the page that link was on), so now it indexes your content as belonging to that URL.

  10. Bingo, a brand new page is created (nevermind that it does not exist IRL, to Googlebot it does)

  11. PR for the new page is assigned later in the process. My best bet: This is an initial calculation that is done something like: PR for the page holding the link less one.

  12. Some other Googlebot finds your page at your right URL and indexes it.

  13. When both pages arrive at the reception of the "index" they are spotted by the "duplicate filter" as it is discovered that they are identical.

  14. The "duplicate filter" doesn't know that one of these pages is not a page but just a link. It has two URLs and identical content, so this is a piece of cake: Let the best page win. The other disappears.


So, essentially, by doing the right thing (interpret a 302 as per the RFC) Google allows another webmaster to convince it's bot that your website is nothing but a temporary holding place for content.

Further, this leads to creation of pages in the index that are not real pages. And, you can do nothing about it.

[edited by: claus at 3:45 pm (utc) on Mar. 9, 2005]


I'm sure this will be rectified to some extent soon, as there is an explosion of publicity about it on the web now, and Yahoo have had the problem fixed for a long time now. MSN seems to be succeptable to it also, but not as much as G.

Sunday, March 06, 2005

Always redirect non www urls, rather than alias them.

I've just noticed that our site is showing up on the search engines as two listings, one for the url: http://ourdomain.com.au and a separate one for http://www.ourdomain.com.au This is interesting becasue our site operates on a remotely hosted Microsoft IIS server, and when I asked them to make the non www version work, I noticed that the address in the browser didn't change when I entered it in without the www prefix. I thought "no worries" and preceeded to ensure that all links on the front page specified the full path with the www prefix. I also try to make all external links into the site include the www prefix aswell, and there seemed to be no problem, but now I've noticed this other listing for our site, and it's ranking higher than us, but is not indexed. I've just asked the web server guys to put the IIS equivalent of a 301 redirect in instead. I've noticed that this is what Google and many other large sites seem to do.

Friday, March 04, 2005

Saturday Motorsport, working in the shop...

Well it's a cruisy Saturday, and Im working my weekend job (in a shop) watching the racing cars on TV. The Pizza place next door opens in an hour and I can't wait. I guess another cigarette will have to do:S
The Pizza place next door is great, I get one every weekend. It's not a Dominos, or a Pizza Hut or anything like that, it's a little place run by an Italian lady. The toppings are fresh and it's %100 better than anything else I recon. True Italian pizza tends to have a thinner crust and they have less ingredients in the topping. I prefer it.

Wednesday, March 02, 2005

Top search terms. These are some of the top terms searched for on the net, courtesy of Wordtracker

Wordtracker provides a monthy list of popular search terms. Here is the list from this month. Some of those terms are a bit of a worry!

The top 300 surge report
These are the top 300 queries from the last 24 hours, which
represents the complete queries from the largest Metacrawlers
on the web (Metacrawler/Dogpile etc..) for the last 24 hours.
Nos. Count Keyword
===========================================
1 5012 sex
2 4203 pizza coupons
3 4202 oscar dresses
4 4046 porn
5 3968 paris hilton
6 3325 giant panda
7 2944 pussy
8 2758 tax help
9 2579 funny commercials
10 2489 milf
11 2357 dell laptops
12 2348 hentai
13 2061 free porn
14 1991 google
15 1977 oscars
16 1901 ebay
17 1855 lesbians
18 1811 coffee mugs
19 1762 green day
20 1709 yahoo
21 1709 boobs
22 1705 britney spears
23 1680 pope health
24 1671 jenna jameson
25 1524 eminem
26 1493 carmen electra
27 1479 tivo
28 1419 tits
29 1393 preteen
30 1367 pamela anderson
31 1363 incest
32 1337 jennifer lopez
33 1306 girls
34 1301 green day cd
35 1297 lindsay lohan
36 1292 btk caught
37 1237 anime
38 1229 50 cent
39 1183 preteen models
40 1169 lolita
41 1163 playboy
42 1154 hot girls
43 1128 girls gone wild
44 1105 literotica
45 1099 nude
46 1089 vagina
47 1083 lesbian
48 1063 big tits
49 1024 mapquest
50 958 candy shop
51 941 xxx
52 941 girls kissing
53 929 beastiality
54 925 hilary duff
55 918 jessica simpson
56 902 ass
57 900 thongs
58 895 usher
59 870 games
60 869 schnappi
61 864 anal
62 858 nudist
63 818 path client
64 799 lesbian sex
65 791 blowjob
66 788 lesbian orgy
67 778 jesse mccartney
68 776 thong
69 774 rape
70 771 weather
71 766 fucking
72 765 yahoo.com
73 751 teen
74 750 teen sex
75 749 panties
76 744 dog
77 743 rich girl
78 742 ask jeeves
79 740 naked
80 739 search engines
81 730 lingerie
82 724 dictionary
83 712 penis
84 705 bikini
85 698 gwen stefani
86 695 naked girls
87 689 breasts
88 682 cars
89 679 dashboard confessional
90 676 masturbation
91 675 dogs
92 674 cum
93 666 fuck
94 666 literotica.com
95 665 angelina jolie
96 664 teen girls
97 662 simple plan
98 651 anal sex
99 640 upskirt
100 635 inuyasha
101 627 gay sex
102 626 orgasm
103 625 brooke burke
104 624 camel toe
105 618 kelly clarkson
106 617 topless
107 617 christina aguilera
108 604 blow job
109 602 naked women
110 587 linkin park
111 582 blowjobs
112 578 hotmail
113 577 nelly
114 574 ciara
115 573 milfs
116 570 hot
117 568 like toy soldiers
118 563 babes
119 562 ashlee simpson
120 561 milf hunter
121 560 sex stories
122 559 voyeur
123 558 blink 182
124 558 dildo
125 558 bang bus
126 558 wet pussy
127 557 bondage
128 554 erotic stories
129 552 katie price
130 552 free porn movies
131 550 nude girls
132 549 animal sex
133 549 btk
134 546 lolitas
135 545 home depot
136 544 teen lesbian
137 543 jokes
138 540 sex positions
139 539 ebay.com
140 534 beyonce
141 534 pokemon hentai
142 533 oral sex
143 526 the killers
144 523 free sex stories
145 520 lyrics
146 518 trish stratus
147 517 halle berry
148 516 good charlotte
149 509 big boobs
150 508 hardcore sex
151 506 nascar
152 503 jdm
153 503 dennis rader
154 502 spanking
155 502 gay
156 501 asian
157 500 thehun
158 500 band or song name
159 495 walmart
160 494 ampland
161 493 jill kelly
162 488 porno
163 486 my chemical romance
164 486 pichunter
165 482 dogpile
166 481 metallica
167 480 ludacris
168 472 3 doors down
169 466 female ejaculation
170 465 sexy
171 463 remove drm from wmv
172 462 hotmail.com
173 462 paris hilton nude
174 458 jordan
175 457 teen models
176 455 pam anderson
177 454 lesbian porn
178 453 music
179 451 daddy yankee
180 450 britney spears nude
181 449 evanescence
182 449 google.com
183 446 clitoris
184 445 hardcore
185 444 paris hilton sex tape
186 444 cat
187 442 free games
188 440 torrie wilson
189 439 funny
190 436 briana banks
191 432 korn
192 432 bangbus
193 431 freeones
194 430 prom dresses
195 426 lil kim
196 426 women
197 426 canadian guestbook
198 425 worldsex.com
199 423 young girls
200 422 tsunami
201 422 movies
202 421 lowes
203 420 teens
204 420 lose my breath
205 419 britney spears naked
206 419 mr. brightside
207 418 orgy
208 418 creampie
209 417 thehun.com
210 416 canadian guest
211 411 simpsons
212 411 napoleon dynamite
213 410 maps
214 409 mariah carey
215 407 www.thehun.com
216 406 slipknot
217 406 beautiful soul
218 405 13 year old girl breast
219 404 deep throat
220 403 clit
221 401 best buy
222 400 song lyrics
223 399 mature
224 396 airline tickets
225 392 xnxx
226 392 tara reid
227 391 avril lavigne
228 391 academy awards
229 390 voyeurweb.com
230 389 paris hilton video
231 388 aria giovanni
232 387 icuii activation code
233 387 nipples
234 387 www.yahoo.com
235 387 nude celebs
236 384 bulma
237 384 love
238 383 thumbzilla
239 383 yahoo mail
240 380 ashanti
241 379 travel
242 378 pokemon
243 378 kelly blue book
244 377 cunt
245 376 bw017.jpg
246 376 preteens
247 376 jenna haze
248 376 shemale
249 376 fisting
250 376 blow jobs
251 375 msn
252 366 amazon.com
253 366 madonna
254 366 kids
255 366 michael jackson
256 364 preteen model galleries
257 361 voyeurweb
258 361 cfnm
259 360 nude women
260 359 irs
261 357 boats
262 356 swingers
263 356 teen thongs
264 355 69
265 355 jennifer love hewitt
266 351 tommys bookmarks
267 351 map quest
268 350 family sex
269 350 free online games
270 349 fafsa
271 349 worldsex
272 348 college girls
273 348 u2
274 348 gang bang
275 347 jennifer aniston
276 346 p
277 346 k-maro
278 346 used cars
279 346 models
280 346 sex toys
281 345 nudists
282 344 bestiality
283 343 spybot
284 342 anime girls
285 340 manga
286 340 tera patrick
287 338 nightwish
288 337 preteen sex
289 337 canadian homeless homeless guestbook
290 336 free music downloads
291 336 shannon elizabeth
292 335 girl
293 334 horses
294 333 ray charles
295 333 elvis presley
296 332 mario
297 331 greenday
298 329 jenna
299 328 sears
300 328 cock

The top 200 long-term keyword report
Our current database contains 368,159,632 search terms
and represents the complete queries from the largest
metacrawlers on the web (Metacrawler/Dogpile etc.) for the
last 60 days (2 months).

Nos. Count Keyword
===========================================
1 302966 sex
2 225620 porn
3 183624 pussy
4 141514 google
5 130414 ebay
6 128080 tits
7 121648 hentai
8 120520 yahoo
9 115379 boobs
10 102071 milf
11 101694 paris hilton
12 99866 health
13 97178 jokes
14 88698 free porn
15 85461 nude
16 80457 girls
17 75322 mapquest
18 73855 yahoo.com
19 71273 games
20 68648 thong
21 68098 dictionary
22 67177 penis
23 66686 britney spears
24 66442 jenna jameson
25 62891 playboy
26 61505 lesbians
27 61188 vagina
28 59235 blowjob
29 59018 naked
30 58652 search engines
31 57734 pamela anderson
32 57446 hotmail
33 57300 anime
34 57176 xxx
35 56755 april fools
36 55450 bikini
37 53925 lyrics
38 53162 lingerie
39 52031 literotica
40 51424 breasts
41 50173 nudist
42 50157 panties
43 49937 carmen electra
44 49898 maps
45 49406 American Idol
46 48776 weather
47 47670 ask jeeves
48 47391 dogs
49 47172 hotmail.com
50 47026 thongs
51 45410 tattoos
52 45397 ass
53 45302 anal
54 45144 89.com
55 44846 incest
56 43869 fucking
57 43451 lolita
58 42765 prom dresses
59 42308 lesbian
60 41892 google.com
61 41483 naked girls
62 41127 porno
63 40260 cars
64 39868 bondage
65 39541 big tits
66 39492 eminem
67 39442 ebay.com
68 38580 fuck
69 38246 parent
70 38200 cunt
71 37609 cum
72 37357 jobs
73 37183 sublime directory
74 36104 Britney Spears
75 36065 spanking
76 35859 masturbation
77 35835 music
78 35512 wallpaper
79 35331 girls gone wild
80 35193 voyeurweb
81 34840 erotic stories
82 34551 sex stories
83 34274 naked women
84 34154 sexy
85 33979 upskirt
86 33918 nude girls
87 33686 Free Games
88 33355 http:
89 33242 home depot
90 33130 spybot
91 33010 american idol
92 32860 jessica simpson
93 32627 flowers
94 32513 kazaa lite
95 32301 voyeur
96 32151 ampland
97 31810 lolitas
98 31789 author
99 31767 song lyrics
100 31412 janet jackson
101 31162 beastiality
102 31074 www.yahoo.com
103 31062 free games
104 30852 William Hung
105 30753 women
106 30332 anal sex
107 30271 babes
108 29910 topless
109 29670 msn emotions
110 29353 recipes
111 29335 preteen
112 29004 baby names
113 28808 airline tickets
114 28725 nudes
115 28671 literotica.com
116 28659 amazon.com
117 28640 blow job
118 27648 funny
119 27322 free sex stories
120 27317 Camel Spiders
121 26999 msn.com
122 26968 blink 182
123 26954 linkin park
124 26789 hair styles
125 26722 orlando bloom
126 26549 usher
127 26465 white pages
128 26315 oral sex
129 26246 movies
130 26240 aol.com
131 26222 preteen models
132 26160 breast
133 26134 kids
134 26133 msn
135 26101 kelly blue book
136 25823 teen
137 25791 priceless
138 25525 clip art
139 25303 southwest airlines
140 25299 quotes
141 25238 funny pictures
142 25201 driving directions
143 25166 walmart
144 24991 www.google.com
145 24901 used cars
146 24877 dildo
147 24777 birthdays
148 24652 irs
149 24306 johnny depp
150 24289 nipples
151 24280 rape
152 24186 dragons
153 23914 map quest
154 23900 pantyhose
155 23868 real estate
156 23767 kazaa
157 23683 nude women
158 23632 clit
159 23505 50 cent
160 23503 orgasm
161 23481 poems
162 23386 spring break
163 23155 Recipes
164 23113 free clip art
165 22994 Baby Names
166 22821 hot girls
167 22793 christina aguilera
168 22633 shemale
169 22548 paris hilton video
170 22515 thumbzilla
171 22499 gay
172 22460 Hair Styles
173 22252 mardi gras
174 22246 pregnancy
175 22106 tattoo
176 22049 yellow pages
177 21981 swingers
178 21972 south beach diet
179 21875 lesbian sex
180 21779 lowes
181 21770 dog
182 21751 tommys bookmarks
183 21722 the hun
184 21659 fisting
185 21584 hairstyles
186 21583 freeones
187 21537 inuyasha
188 21536 Tattoos
189 21472 adult
190 21308 gay sex
191 21259 zip codes
192 21157 big boobs
193 21141 blowjobs
194 20937 boob
195 20909 sears
196 20886 nudists
197 20831 camel toe
198 20826 cats
199 20815 incest stories
200 20716 www.domai.com