phpBB2Refugees.com Logo
Not affiliated with or endorsed by the phpBB Group

Register •  Login 

Continue the legacy...

Welcome to all phpBB2 Refugees!Wave Smilie

This site is intended to continue support for the legacy 2.x line of the phpBB2 bulletin board package. If you are a fan of phpBB2, please, by all means register, post, and help us out by offering your suggestions. We are primarily a community and support network. Our secondary goal is to provide a phpBB2 MOD Author and Styles area.

Scraper and other bot/bot net attacks


 
Search this topic... | Search phpBB2 Discussion... | Search Box
Register or Login to Post    Index » phpBB2 Discussion  Previous TopicPrint TopicNext Topic
Author Message
JLA
Board Member



Joined: 30 Apr 2009

Posts: 309
Location: U.S.A


flag
PostPosted: Sun Feb 07, 2010 5:12 pm 
Post subject: Scraper and other bot/bot net attacks

So how about a discussion regarding site scraper, posting bots, hackers and other bot net attacks?

Since 2004 we have accumulated tons of data to determine several patterns in site scrapers, posting bot, hack attempts and other forms of PHPBB attacks by bot nets.

Anyone who looks deeply into their server logs will understand that

1. Site content scrapers are a HUGE problem. There are tons of organizations and individuals that are out there who's main purpose is to scrape your site for every bit of data/content they can and sell it or use it for their own purposes. The problem for you is it costs you bandwidth and whatever time and money you have spent producing your content. What is very interesting about this is the type of organizations and their purpose in scraping your site. Once you have an understanding of this you will find there is ever a bigger reason to be concerned about preventing this sort of activity on your site.

2. Posting bots everyone is well aware of. These bots make attempts to automatically post spam to your site. The bots can sometimes overwhelm your site and fill your forum with so much junk it is useless.

3. Hackers - depending on the size of your site you will find that there are hundreds, to thousands or more attempts to hack your site in some way shape or form ever single day. These range from script kiddies using easy to find code to attempt to use some old exploit against your site to more advanced individuals making very pin-pointed attempts to breach your site security. Their purpose? Some just to see if they can do it others for various other more malicious reasons.

4. Bot Net Attacks. A single individual or organization controls thousands to millions of zombie computers from a single point to either DDOS your site or perform the tasks in #1, #2, #3 or all of the above.

So how can these sort of things be dealt with? Well since PHPBB2 is a pretty mature platform, there are alot of scripts/code out there to help the script kiddies and even other bigger (well known) organizations in their attempts to do the things to your site mentioned above.

Posting bots are easier to deal with. There are tons of mods to deal with automated posting to your site. Always keep in mind that there is no single magic solution to dealing with posting bots. You must use a multi-layered approach.

Site scrapers, hackers and bot net attacks are another issue all together. Some suggest using some of the phpbb security mods that have been written over the years but even these mods have had "security" issues of their own. You also have to take into account site/database performance issues when using many of these security mods. Keep in mind that many of the older mods written may have been countered by newer malicious scripts which in effect render those older mods useless.

_________________
http://www.jlaforums.com
Back to top
Dog Cow
Board Member



Joined: 18 Nov 2008

Posts: 378


flag
PostPosted: Mon Feb 08, 2010 2:36 pm 
Post subject: Re: Scraper and other bot/bot net attacks

This is my solution: http://phpbb2refugees.com/viewtopic.php?t=338

But as I recall, you had some problems with it. Overall, most of these problems should be dealt with at the web server level, not the application.

_________________
Moof!
Lincoln's Tomb, Oak Ridge Cemetery, Springfield IL Mac 512K Blog Mac GUI
Back to top
Holger
Board Member



Joined: 19 Jan 2009

Posts: 509
Location: Hanover


flag
PostPosted: Tue Feb 09, 2010 1:55 pm 
Post subject: Re: Scraper and other bot/bot net attacks

I have to say I have seen a "drastical reduction" of attacks by hackers and bots in the past year.
They are not interested in phpBB2 anymore.
I have taken som countermeasures against hackers and bots and my forum is up since 2003, but the only times we have been hacked (maybe two or three times) was when the hoster was "weak", not phpBB2.
No hackings since four years now.


On the other hand the scrapers are still the same. Interesting, what have you found out about them?
Back to top
Dog Cow
Board Member



Joined: 18 Nov 2008

Posts: 378


flag
PostPosted: Wed Feb 10, 2010 8:32 pm 
Post subject: Re: Scraper and other bot/bot net attacks

Holger wrote:
Interesting, what have you found out about them?
That they can be stopped with some HTML comments. Check my site's source, in the head, and then robots.txt.
_________________
Moof!
Lincoln's Tomb, Oak Ridge Cemetery, Springfield IL Mac 512K Blog Mac GUI
Back to top
JLA
Board Member



Joined: 30 Apr 2009

Posts: 309
Location: U.S.A


flag
PostPosted: Thu Feb 11, 2010 4:41 am 
Post subject: Re: Scraper and other bot/bot net attacks

Dog Cow wrote:
This is my solution: http://phpbb2refugees.com/viewtopic.php?t=338

But as I recall, you had some problems with it. Overall, most of these problems should be dealt with at the web server level, not the application.


I think we had tried this at one time and it didn't work out for us. Probably due to the operating environment the site works off of.

_________________
http://www.jlaforums.com
Back to top
JLA
Board Member



Joined: 30 Apr 2009

Posts: 309
Location: U.S.A


flag
PostPosted: Thu Feb 11, 2010 4:42 am 
Post subject: Re: Scraper and other bot/bot net attacks

Holger wrote:
I have to say I have seen a "drastical reduction" of attacks by hackers and bots in the past year.
They are not interested in phpBB2 anymore.
I have taken som countermeasures against hackers and bots and my forum is up since 2003, but the only times we have been hacked (maybe two or three times) was when the hoster was "weak", not phpBB2.
No hackings since four years now.


On the other hand the scrapers are still the same. Interesting, what have you found out about them?


We have seen an incremental increase of activity each month on all four items as mentioned in the OP. This has been going on since 2005/2006.

The scrapers are pretty bad and getting more advanced. Apparently there are quite a few organizations such as bran***d***watch/brand***dim**ensions/web***sen**se, as well as many of the Russians, Chinese, Arabs and French, etc that hit sites pretty hard. When they are blocked at the organizational IP blocks they then start resorting to using bot nets and other similar methods. Apparently scraping content and reselling it in various forms is big business.

Just have a look at your server logs and start digging into what is going on and you'll see what we mean.

_________________
http://www.jlaforums.com
Back to top
Holger
Board Member



Joined: 19 Jan 2009

Posts: 509
Location: Hanover


flag
PostPosted: Thu Feb 11, 2010 8:14 am 
Post subject: Re: Scraper and other bot/bot net attacks

JLA wrote:
Just have a look at your server logs and start digging into what is going on and you'll see what we mean.

Well, I have not noticed other then that there are tries to fetch random files, like "users.mdb" and such non-existing files.
But the activities are rather low.

What exactly should I look after in the log-files?
Can you give me an example? I will sure find SOMETHING, but I do not know what to search for.
Back to top
Dog Cow
Board Member



Joined: 18 Nov 2008

Posts: 378


flag
PostPosted: Thu Feb 11, 2010 4:43 pm 
Post subject: Re: Scraper and other bot/bot net attacks

Holger wrote:

What exactly should I look after in the log-files?
Can you give me an example? I will sure find SOMETHING, but I do not know what to search for.

Just to get you started, do a search for all HTTP/1.0 connections which aren't either Yahoo or Twiceler.

Here's what a scrape attack looks like:
(you might want to copy this into a text editor to read it more easily.
Code:

82.166.235.133 - - [02/Feb/2010:07:28:43 -0500] "GET / HTTP/1.1" 200 2734 "" "Microsoft Internet Explorer/3.3 (Windows NT 2.6; )"
82.166.235.133 - - [02/Feb/2010:07:28:44 -0500] "GET / HTTP/1.1" 200 2734 "" "Opera/1.9 (Windows 2000 1.1; )"
82.166.235.133 - - [02/Feb/2010:07:28:44 -0500] "GET /foobaz.php HTTP/1.1" 200 - "" "ia_archiver/5.5 (OSX 2.5; )"
82.166.235.133 - - [02/Feb/2010:07:28:45 -0500] "GET /index.php HTTP/1.1" 403 87 "" "Microsoft Internet Explorer/4.6 (Windows 2000 6.3; )"
82.166.235.133 - - [02/Feb/2010:07:28:45 -0500] "GET /mygui/join.php HTTP/1.1" 403 87 "" "ia_archiver/6.3 (Windows 3.1; )"
82.166.235.133 - - [02/Feb/2010:07:28:46 -0500] "GET /search HTTP/1.1" 301 237 "" "ia_archiver/1.6 (Windows 2000 3.8; )"
82.166.235.133 - - [02/Feb/2010:07:28:46 -0500] "GET /search/ HTTP/1.1" 403 87 "http://www.macgui.com:80/search" "ia_archiver/1.6 (Windows 2000 3.8; )"
82.166.235.133 - - [02/Feb/2010:07:28:47 -0500] "GET /infobooth/faq HTTP/1.1" 403 87 "" "Microsoft Internet Explorer/4.3 (OSX 4.5; )"
82.166.235.133 - - [02/Feb/2010:07:28:47 -0500] "GET /cc HTTP/1.1" 301 233 "" "Mozilla/5.1 (Windows XP 2.5; )"
82.166.235.133 - - [02/Feb/2010:07:28:48 -0500] "GET /cc/ HTTP/1.1" 403 87 "http://www.macgui.com:80/cc" "Mozilla/5.1 (Windows XP 2.5; )"
82.166.235.133 - - [02/Feb/2010:07:28:48 -0500] "GET /community HTTP/1.1" 301 240 "" "ia_archiver/8.6 (OSX 6.5; )"
82.166.235.133 - - [02/Feb/2010:07:28:49 -0500] "GET /community/ HTTP/1.1" 403 87 "http://www.macgui.com:80/community" "ia_archiver/8.6 (OSX 6.5; )"
82.166.235.133 - - [02/Feb/2010:07:28:49 -0500] "GET /news HTTP/1.1" 301 235 "" "Mozilla/6.9 (Windows NT 4.8; )"
82.166.235.133 - - [02/Feb/2010:07:28:50 -0500] "GET /news/ HTTP/1.1" 403 87 "http://www.macgui.com:80/news" "Mozilla/6.9 (Windows NT 4.8; )"
82.166.235.133 - - [02/Feb/2010:07:28:50 -0500] "GET /emc/qm.php HTTP/1.1" 403 87 "" "Opera/7.7 (Windows 2.0; )"
82.166.235.133 - - [02/Feb/2010:07:28:51 -0500] "GET /plaza HTTP/1.1" 403 87 "" "Microsoft Internet Explorer/7.3 (Windows 7.6; )"
82.166.235.133 - - [02/Feb/2010:07:28:51 -0500] "GET /market HTTP/1.1" 301 237 "" "Opera/4.5 (Windows 2000 6.9; )"
82.166.235.133 - - [02/Feb/2010:07:28:52 -0500] "GET /market/ HTTP/1.1" 403 87 "http://www.macgui.com:80/market" "Opera/4.5 (Windows 2000 6.9; )"
82.166.235.133 - - [02/Feb/2010:07:28:52 -0500] "GET /suburb HTTP/1.1" 301 237 "" "Opera/8.1 (Windows XP 2.4; )"
82.166.235.133 - - [02/Feb/2010:07:28:53 -0500] "GET /suburb/ HTTP/1.1" 403 87 "http://www.macgui.com:80/suburb" "Opera/8.1 (Windows XP 2.4; )"
82.166.235.133 - - [02/Feb/2010:07:28:53 -0500] "GET /infobooth HTTP/1.1" 403 87 "" "Opera/2.7 (OSX 4.2; )"
82.166.235.133 - - [02/Feb/2010:07:28:54 -0500] "GET /fountain HTTP/1.1" 403 87 "" "Mozilla/5.6 (Linux 7.0; )"
82.166.235.133 - - [02/Feb/2010:07:28:54 -0500] "GET /infobooth/directory HTTP/1.1" 403 87 "" "Mozilla/3.2 (Windows 2000 4.6; )"
82.166.235.133 - - [02/Feb/2010:07:28:55 -0500] "GET /downloads HTTP/1.1" 301 240 "" "Microsoft Internet Explorer/3.5 (Windows 3.9; )"
82.166.235.133 - - [02/Feb/2010:07:28:55 -0500] "GET /downloads/ HTTP/1.1" 403 87 "http://www.macgui.com:80/downloads" "Microsoft Internet Explorer/3.5 (Windows 3.9; )"
82.166.235.133 - - [02/Feb/2010:07:28:56 -0500] "GET /vault HTTP/1.1" 403 87 "" "ia_archiver/4.7 (Linux 6.3; )"
82.166.235.133 - - [02/Feb/2010:07:28:57 -0500] "GET /downloads/?file_id=23923 HTTP/1.1" 403 87 "" "Mozilla/7.1 (Windows NT 6.9; )"
82.166.235.133 - - [02/Feb/2010:07:28:57 -0500] "GET /downloads/?file_id=23922 HTTP/1.1" 403 87 "" "Opera/7.1 (Windows 3.9; )"
82.166.235.133 - - [02/Feb/2010:07:28:58 -0500] "GET /downloads/?file_id=23921 HTTP/1.1" 403 87 "" "Microsoft Internet Explorer/7.2 (Linux 3.5; )"
82.166.235.133 - - [02/Feb/2010:07:29:00 -0500] "GET /downloads/?file_id=23920 HTTP/1.1" 403 87 "" "Mozilla/1.9 (Windows 2000 1.5; )"
82.166.235.133 - - [02/Feb/2010:07:29:00 -0500] "GET /downloads/?file_id=23919 HTTP/1.1" 403 87 "" "Opera/8.2 (OSX 4.5; )"
82.166.235.133 - - [02/Feb/2010:07:29:01 -0500] "GET /news/article.php?t=238 HTTP/1.1" 403 87 "" "Microsoft Internet Explorer/3.2 (OSX 3.9; )"
82.166.235.133 - - [02/Feb/2010:07:29:01 -0500] "GET /news/article.php?t=237 HTTP/1.1" 403 87 "" "Mozilla/8.6 (Windows XP 4.5; )"
82.166.235.133 - - [02/Feb/2010:07:29:02 -0500] "GET /news/article.php?t=236 HTTP/1.1" 403 87 "" "Opera/2.6 (Windows NT 7.2; )"
82.166.235.133 - - [02/Feb/2010:07:29:02 -0500] "GET /news/article.php?t=235 HTTP/1.1" 403 87 "" "ia_archiver/2.0 (Linux 4.2; )"
82.166.235.133 - - [02/Feb/2010:07:29:03 -0500] "GET /news/article.php?t=234 HTTP/1.1" 403 87 "" "Mozilla/8.0 (Windows 2000 7.1; )"
82.166.235.133 - - [02/Feb/2010:07:29:05 -0500] "GET /forums/cafe/do-you-shop-online/t.1588_1 HTTP/1.1" 403 87 "" "ia_archiver/6.9 (Windows XP 7.0; )"
82.166.235.133 - - [02/Feb/2010:07:29:05 -0500] "GET /forums/cafe/what-is-your-weather-like/t.1531_20 HTTP/1.1" 403 87 "" "Mozilla/2.9 (Windows 2000 3.6; )"
82.166.235.133 - - [02/Feb/2010:07:29:06 -0500] "GET /forums/cafe/im/t.98_193 HTTP/1.1" 403 87 "" "Mozilla/7.8 (Linux 1.0; )"
82.166.235.133 - - [02/Feb/2010:07:29:06 -0500] "GET /forums/hardware-hall/ibook-questions/t.1587_1 HTTP/1.1" 403 87 "" "ia_archiver/1.5 (Windows 2000 3.5; )"
82.166.235.133 - - [02/Feb/2010:07:29:07 -0500] "GET /forums/cafe/last-letter-game/t.130_171 HTTP/1.1" 403 87 "" "Microsoft Internet Explorer/4.4 (Windows XP 7.5; )"
82.166.235.133 - - [02/Feb/2010:07:29:07 -0500] "GET /forums HTTP/1.1" 301 237 "" "Mozilla/3.3 (Windows 2000 5.9; )"
82.166.235.133 - - [02/Feb/2010:07:29:08 -0500] "GET /forums/ HTTP/1.1" 403 87 "http://www.macgui.com:80/forums" "Mozilla/3.3 (Windows 2000 5.9; )"
82.166.235.133 - - [02/Feb/2010:07:29:08 -0500] "GET /blogs/?e=348 HTTP/1.1" 403 87 "" "ia_archiver/1.9 (Linux 2.4; )"
82.166.235.133 - - [02/Feb/2010:07:29:09 -0500] "GET /blogs/?e=347 HTTP/1.1" 403 87 "" "Opera/8.2 (OSX 5.5; )"
82.166.235.133 - - [02/Feb/2010:07:29:09 -0500] "GET /blogs/?e=346 HTTP/1.1" 403 87 "" "ia_archiver/4.2 (Windows 2000 6.1; )"
82.166.235.133 - - [02/Feb/2010:07:29:10 -0500] "GET /blogs/?e=345 HTTP/1.1" 403 87 "" "Mozilla/5.8 (Windows NT 2.4; )"
82.166.235.133 - - [02/Feb/2010:07:29:10 -0500] "GET /blogs/?e=344 HTTP/1.1" 403 87 "" "Mozilla/2.6 (Windows XP 2.5; )"
82.166.235.133 - - [02/Feb/2010:07:29:11 -0500] "GET /blogs HTTP/1.1" 301 236 "" "ia_archiver/5.2 (OSX 3.3; )"
82.166.235.133 - - [02/Feb/2010:07:29:11 -0500] "GET /blogs/ HTTP/1.1" 403 87 "http://www.macgui.com:80/blogs" "ia_archiver/5.2 (OSX 3.3; )"

Notice two things:
1.) The spider starts at index.php (home page) and accesses each URL in order it appears in your HTML code
2.) Once the spider hits foobaz.php, all successive page requests return a 403

If the spider didn't hit foobaz.php, then it would have been stopped by my page requests per second limiter.


Here's a second example of a really poorly-written spider. Notice how it isn't making any legal HTTP requests, hence the 400 errors being returned by Stupid Bots:
Code:

74.86.153.132 - - [06/Feb/2010:22:50:52 -0500] "GET / HTTP/1.1" 200 9400 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:54 -0500] "GET //\"http://www.macgui.com/news/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:54 -0500] "GET //\"http://www.macgui.com/mygui/join.php HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:54 -0500] "GET //\"http://www.macgui.com/foobaz.php HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:54 -0500] "GET //\"http://www.macgui.com/plaza/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:54 -0500] "GET //\"http://www.macgui.com/infobooth/faq/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:54 -0500] "GET //\"http://www.macgui.com/community/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:54 -0500] "GET //\"http://www.macgui.com/emc/qm.php HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:54 -0500] "GET //\"http://www.macgui.com/index.php HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:54 -0500] "GET //\"http://www.macgui.com/cc/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:54 -0500] "GET //\"http://www.macgui.com/search/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:59 -0500] "GET //\"http://www.macgui.com/fountain/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:00 -0500] "GET //\"http://www.macgui.com/vault/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:59 -0500] "GET //\"http://www.macgui.com/infobooth/directory/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:00 -0500] "GET //\"http://www.macgui.com/infobooth/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:00 -0500] "GET //\"http://www.macgui.com/mygui/join.php HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:50:59 -0500] "GET //\"http://www.macgui.com/market/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:00 -0500] "GET //\"http://www.macgui.com/downloads/?file_id=23922 HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:00 -0500] "GET //\"http://www.macgui.com/suburb/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:00 -0500] "GET //\"http://www.macgui.com/downloads/?file_id=23923 HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:00 -0500] "GET //\"http://www.macgui.com/downloads/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:01 -0500] "GET //\"http://www.macgui.com/downloads/?file_id=23921 HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:01 -0500] "GET //\"http://www.macgui.com/downloads/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:01 -0500] "GET //\"http://www.macgui.com/downloads/?file_id=23919 HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:01 -0500] "GET //\"http://www.macgui.com/news/article.php?t=238 HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:01 -0500] "GET //\"http://www.macgui.com/news/article.php?t=236 HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:01 -0500] "GET //\"http://www.macgui.com/news/ HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:01 -0500] "GET //\"http://www.macgui.com/news/article.php?t=237 HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:01 -0500] "GET //\"http://www.macgui.com/downloads/?file_id=23920 HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:01 -0500] "GET //\"http://www.macgui.com/news/article.php?t=235 HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"
74.86.153.132 - - [06/Feb/2010:22:51:01 -0500] "GET //\"http://www.macgui.com/news/article.php?t=234 HTTP/1.1" 400 15 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.4) Gecko/20070625 Ubuntu/7.10 (gutsy) Firefox/2.0.0.4"

It was likely written by some amateur.

_________________
Moof!
Lincoln's Tomb, Oak Ridge Cemetery, Springfield IL Mac 512K Blog Mac GUI
Back to top
JLA
Board Member



Joined: 30 Apr 2009

Posts: 309
Location: U.S.A


flag
PostPosted: Fri Feb 12, 2010 4:03 pm 
Post subject: Re: Scraper and other bot/bot net attacks

Holger wrote:
JLA wrote:
Just have a look at your server logs and start digging into what is going on and you'll see what we mean.

Well, I have not noticed other then that there are tries to fetch random files, like "users.mdb" and such non-existing files.
But the activities are rather low.

What exactly should I look after in the log-files?
Can you give me an example? I will sure find SOMETHING, but I do not know what to search for.


A good place to start is to look at visitors that are requesting pages (viewtopic.php) but none of the images or other items on your page at any time. With the exception of good bots like Google, MSN, Yahoo, etc - these visitors have absolutely no benefit to you and most likely are scraping your content.

_________________
http://www.jlaforums.com
Back to top
Holger
Board Member



Joined: 19 Jan 2009

Posts: 509
Location: Hanover


flag
PostPosted: Fri Feb 12, 2010 4:08 pm 
Post subject: Re: Scraper and other bot/bot net attacks

JLA wrote:
A good place to start is to look at visitors that are requesting pages (viewtopic.php) but none of the images or other items on your page at any time. With the exception of good bots like Google, MSN, Yahoo, etc - these visitors have absolutely no benefit to you and most likely are scraping your content.

Good point!
Will check that!
Back to top
JLA
Board Member



Joined: 30 Apr 2009

Posts: 309
Location: U.S.A


flag
PostPosted: Fri Feb 12, 2010 4:21 pm 
Post subject: Re: Scraper and other bot/bot net attacks

Holger wrote:
JLA wrote:
A good place to start is to look at visitors that are requesting pages (viewtopic.php) but none of the images or other items on your page at any time. With the exception of good bots like Google, MSN, Yahoo, etc - these visitors have absolutely no benefit to you and most likely are scraping your content.

Good point!
Will check that!


A good thing to do is to find a good real time log parser that allows you to watch your traffic in real time. Keep in mind this is NOT the same as using other tracking services which are cookie based which are easily passed over (hence you do not see the real traffic). The real time log parser lets you see the actual requests - no matter what they are in real time and can be a very useful tool for not only managing your site but gaining a greater understanding of malicious activity against your site that goes on everyday.

_________________
http://www.jlaforums.com
Back to top
Display posts from previous:   
Register or Login to Post    Index » phpBB2 Discussion  Previous TopicPrint TopicNext Topic
Page 1 of 1 All times are GMT
 
Jump to:  

Index • About • FAQ • Rules • Privacy • Search •  Register •  Login 
Not affiliated with or endorsed by the phpBB Group
Powered by phpBB2 © phpBB Group
Generated in 0.0331 seconds using 15 queries. (SQL 0.0025 Parse 0.0009 Other 0.0297)
phpBB Customizations by the phpBBDoctor.com
Template Design by DeLFlo and MomentsOfLight.com Moments of Light Logo