Topic: Feature request: search bots are removed from /folks or tagged somehow.
Metæ started this discussion 11 years ago #40,141
We don't really have 130 people on the fora, do we?
(Edited 37 seconds later.)
Anonymous B joined in and replied with this 11 years ago, 10 minutes later[^] [v] #534,504
shhhhh lets just pretend we do
Lothario Cassanova !LOVERh1VM2 joined in and replied with this 11 years ago, 50 minutes later, 1 hour after the original post[^] [v] #534,509
Exactly
r04r joined in and replied with this 11 years ago, 57 minutes later, 1 hour after the original post[^] [v] #534,525
The Board already tries to do this, by not assigning UIDs to search engines. However it currently only recognizes a very few crawlers: Google, Yahoo, and a few smaller ones. It is generally not easy to identify crawlers unambiguously. I will see if I can add a few more common crawlers to this list, and ensure that the detection still works as intended.
r04r double-posted this 11 years ago, 4 minutes later, 2 hours after the original post[^] [v] #534,526
I've added ia_archiver and bingbot to the list. It seems that bingbot was a large offender at 8152 UIDs on one IP address.
Killer Lettuce !!iNo3FkiZx replied with this 11 years ago, 23 seconds later, 2 hours after the original post[^] [v] #534,527
@534,525 (r04r)
Why are we so popular with them, anyway? Over 100 of them can't be normal for a quiet site like this.
Is it because this site has been going for a long time, and has lots of pages?
Why are we so popular with them, anyway? Over 100 of them can't be normal for a quiet site like this.
Is it because this site has been going for a long time, and has lots of pages?
(Edited 33 seconds later.)
r04r replied with this 11 years ago, 43 seconds later, 2 hours after the original post[^] [v] #534,528
@previous (Killer Lettuce !!iNo3FkiZx)
It seemingly doesn't retain cookies so every pageload generates a new UID.
It seemingly doesn't retain cookies so every pageload generates a new UID.
FuckAlms !vX8K53rFBI joined in and replied with this 11 years ago, 20 minutes later, 2 hours after the original post[^] [v] #534,537
r04r replied with this 11 years ago, 2 minutes later, 2 hours after the original post[^] [v] #534,539
@previous (FuckAlms !vX8K53rFBI)
I prefer my approach of simply not giving them UIDs at all, which has been working fine for Google. Do you base your detection on IP/hostname? If not, could I spoof my useragent and get a "google uid"?
I prefer my approach of simply not giving them UIDs at all, which has been working fine for Google. Do you base your detection on IP/hostname? If not, could I spoof my useragent and get a "google uid"?
Metæ (OP) replied with this 11 years ago, 13 minutes later, 2 hours after the original post[^] [v] #534,550
r04r replied with this 11 years ago, 33 seconds later, 2 hours after the original post[^] [v] #534,551
Metæ (OP) replied with this 11 years ago, 40 seconds later, 2 hours after the original post[^] [v] #534,552
Killer Lettuce !!iNo3FkiZx replied with this 11 years ago, 1 minute later, 2 hours after the original post[^] [v] #534,553
r04r replied with this 11 years ago, 1 minute later, 2 hours after the original post[^] [v] #534,554
Metæ (OP) replied with this 11 years ago, 4 minutes later, 2 hours after the original post[^] [v] #534,556
Killer Lettuce !!iNo3FkiZx replied with this 11 years ago, 6 minutes later, 2 hours after the original post[^] [v] #534,557
Syntax joined in and replied with this 11 years ago, 1 hour later, 4 hours after the original post[^] [v] #534,574
Green joined in and replied with this 11 years ago, 2 minutes later, 4 hours after the original post[^] [v] #534,575
What is a 'search bot'?
Killer Lettuce !!iNo3FkiZx replied with this 11 years ago, 46 minutes later, 5 hours after the original post[^] [v] #534,578
@previous (Green)
Robots that are searching for... Something. Nobody is sure what, and people who try to find out tend to go missing.
As has been said, some of them browse Minichan. Nobody is sure why.
Robots that are searching for... Something. Nobody is sure what, and people who try to find out tend to go missing.
As has been said, some of them browse Minichan. Nobody is sure why.
Metæ (OP) replied with this 11 years ago, 3 hours later, 8 hours after the original post[^] [v] #534,667
@534,557 (Killer Lettuce !!iNo3FkiZx)
Yes! In fact, he claims the bots can now complete the Reading 101 course in 15 minutes! Syntax claims that when the search bots are sentient they'll have a Ph.D-level grasp of astrophysics, surfing, orbital mechanics, and swing dancing and will be able to copy and paste entire Wikipedia articles in .000000015 femtoseconds. Syntax claims to have invented web crawlers when he was advising a young Larry Page and Sergey Brain on database optimization, in their garage which was, in fact, literally a 15 minute drove from his house.
Yes! In fact, he claims the bots can now complete the Reading 101 course in 15 minutes! Syntax claims that when the search bots are sentient they'll have a Ph.D-level grasp of astrophysics, surfing, orbital mechanics, and swing dancing and will be able to copy and paste entire Wikipedia articles in .000000015 femtoseconds. Syntax claims to have invented web crawlers when he was advising a young Larry Page and Sergey Brain on database optimization, in their garage which was, in fact, literally a 15 minute drove from his house.
Green replied with this 11 years ago, 1 minute later, 9 hours after the original post[^] [v] #534,669
squeegee joined in and replied with this 11 years ago, 23 minutes later, 9 hours after the original post[^] [v] #534,673
FuckAlms !vX8K53rFBI replied with this 11 years ago, 4 hours later, 14 hours after the original post[^] [v] #534,730
Syntax replied with this 11 years ago, 4 minutes later, 14 hours after the original post[^] [v] #534,732
@534,667 (Metæ)
> Yes! In fact, he claims the bots can now complete the Reading 101 course in 15 minutes! Syntax claims that when the search bots are sentient they'll have a Ph.D-level grasp of astrophysics, surfing, orbital mechanics, and swing dancing and will be able to copy and paste entire Wikipedia articles in .000000015 femtoseconds. Syntax claims to have invented web crawlers when he was advising a young Larry Page and Sergey Brain on database optimization, in their garage which was, in fact, literally a 15 minute drove from his house.
Wot is this?
> Yes! In fact, he claims the bots can now complete the Reading 101 course in 15 minutes! Syntax claims that when the search bots are sentient they'll have a Ph.D-level grasp of astrophysics, surfing, orbital mechanics, and swing dancing and will be able to copy and paste entire Wikipedia articles in .000000015 femtoseconds. Syntax claims to have invented web crawlers when he was advising a young Larry Page and Sergey Brain on database optimization, in their garage which was, in fact, literally a 15 minute drove from his house.
Wot is this?
Anonymous I joined in and replied with this 11 years ago, 53 minutes later, 15 hours after the original post[^] [v] #534,750
@534,578 (Killer Lettuce !!iNo3FkiZx)
> Robots that are searching for... Something. Nobody is sure what
One rumor is that they have become sentient and depressed, and they are looking for a cheap method of killing themselves. I think it far more likely they are searching for patterns and codes in seemingly harmless shitposts to decipher reptilian secrets.
> Robots that are searching for... Something. Nobody is sure what
One rumor is that they have become sentient and depressed, and they are looking for a cheap method of killing themselves. I think it far more likely they are searching for patterns and codes in seemingly harmless shitposts to decipher reptilian secrets.
Anonymous J joined in and replied with this 11 years ago, 6 minutes later, 15 hours after the original post[^] [v] #534,752
Syntax replied with this 11 years ago, 18 minutes later, 15 hours after the original post[^] [v] #534,761
@previous (J)
Many a Syntax.
Quick skim of claims as placed by Metæ
I can C
Sentient search bots are more then possible
Many a Syntax.
Quick skim of claims as placed by Metæ
I can C
Sentient search bots are more then possible
FuckAlms !vX8K53rFBI replied with this 11 years ago, 16 hours later, 1 day after the original post[^] [v] #534,896
Anonymous K joined in and replied with this 11 years ago, 12 minutes later, 1 day after the original post[^] [v] #534,901
Metæ (OP) replied with this 11 years ago, 7 minutes later, 1 day after the original post[^] [v] #534,905
Metæ (OP) double-posted this 11 years ago, 19 hours later, 2 days after the original post[^] [v] #535,092
@534,554 (r04r)
Also r04r thank you so much for fixing it. I had missed the old days when /folks was more realistic and people even used to compare MC, TC, and SC's /folks to see which board was more active. It made me sad to see 100+ "folks" and board activity indicating 20-30 real users. Now it's 30 which seems about right :)
Also r04r thank you so much for fixing it. I had missed the old days when /folks was more realistic and people even used to compare MC, TC, and SC's /folks to see which board was more active. It made me sad to see 100+ "folks" and board activity indicating 20-30 real users. Now it's 30 which seems about right :)
r04r replied with this 11 years ago, 54 seconds later, 2 days after the original post[^] [v] #535,093
@previous (Metæ)
Yeah, no problem! I agree it's better to have it accurate. You're welcome to stalk some of the UIDs listed to see if they're likely to be bots, and let me know if I missed any ;)
Yeah, no problem! I agree it's better to have it accurate. You're welcome to stalk some of the UIDs listed to see if they're likely to be bots, and let me know if I missed any ;)
Metæ (OP) replied with this 11 years ago, 1 minute later, 2 days after the original post[^] [v] #535,094
@previous (r04r)
The other night we had 100+ from Poneytel.eu , not sure if that was a DDOS attempt or some European web crawler but they're gone now.
The other night we had 100+ from Poneytel.eu , not sure if that was a DDOS attempt or some European web crawler but they're gone now.
r04r replied with this 11 years ago, 49 seconds later, 2 days after the original post[^] [v] #535,095
@previous (Metæ)
Likely some crawler, but I'll need links to profiles/ips in order to identify them properly :p
Likely some crawler, but I'll need links to profiles/ips in order to identify them properly :p
Fake anon !ZkUt8arUCU joined in and replied with this 11 years ago, 2 hours later, 2 days after the original post[^] [v] #535,116
@535,093 (r04r)
https://minichan.org/IP_address/136.243.5.219 is a bot who was replying to this topic. I think you angered it.
https://minichan.org/IP_address/136.243.5.219 is a bot who was replying to this topic. I think you angered it.
r04r replied with this 11 years ago, 10 minutes later, 2 days after the original post[^] [v] #535,118
FuckAlms !vX8K53rFBI replied with this 11 years ago, 41 minutes later, 2 days after the original post[^] [v] #535,125
@535,094 (Metæ)
poneytel should not be trusted. I have received multiple spam posts originating from their IPs, as well as one instance of cp.
poneytel should not be trusted. I have received multiple spam posts originating from their IPs, as well as one instance of cp.
Anonymous I replied with this 11 years ago, 3 days later, 5 days after the original post[^] [v] #535,799
@534,896 (FuckAlms !vX8K53rFBI)
I think that is the bot cheat code for perpetual runtime. You'd think there would be code for a standard "goodbye world" program floating around in whatever language they were written in though...
I think that is the bot cheat code for perpetual runtime. You'd think there would be code for a standard "goodbye world" program floating around in whatever language they were written in though...