| |
Haven't registered yet? Do it here now!
 |
|
 |
|
JohnnyX
Joined: 27 Oct 2008
Posts: 7
|
|
|
 |
|
 |
|
 Posted: Wed Aug 26, 2009 9:59 pm Post subject: Server has slowed to a crawl due to cronjobs |
 |
|
|
 |
|
 |
|
|
|
|
Hello,
I installed BO a while back and have had no problems. I have now got 559 blogs.
A couple of days ago the server (dedicated) started to slow down, and today it is crawling. My host says there are some cronjobs that are stacked up and and are taking up a lot of resource. It is basically the borssfeeder.php cron that they are saying is at fault. They say some cron jobs have been running for hours.
Now, although you say to set most crons to run every minute, my host does not allow them to run more frequently than every 5 minutes, so I set them up that way and like I say the script has been running perfectly for a couple of months like that.
As they are not finishing in 5 minutes, I have just gone in and changed them to run every 15 minutes, but my server has not speeded up, so I have asked my host to terminate and jobs that have been running for hours.
I'm not sure if that is the right thing to do, and how to stop this happenning again?
Is this due to the number of blogs I have got running on the server?
(I have been with this host for 10 years and never had any problems with a server being slow, even when I used to push far more tgp traffic through it than I do now) |
|
|
|
|
|
|
|
Atanasis
Owner
Joined: 22 May 2004
Posts: 4284
Location: The Net
|
|
|
|
|
|
|
| Posted: Thu Aug 27, 2009 9:30 am Post subject: |
|
|
|
|
|
|
|
|
|
|
yes, it could be like that if you have excessive settings...
tell me these:
1) the value of "settings->exec limits"
2) average crawl time for all blogs
3) average new posts per crawl for all blogs
4) average number of assigned feeds per blog
on these things depend the operation of the borssfeeder. If you for example have 20 feeds assigned to a blog, with pretty frequent crawl rate, that definitely will cause load on the server considering the 560 blogs you have in db.
_________________
Thanks,
Kaktusan
 |
|
|
|
|
|
|
|
JohnnyX
Joined: 27 Oct 2008
Posts: 7
|
|
|
|
|
|
|
| Posted: Fri Aug 28, 2009 12:27 am Post subject: |
|
|
|
|
|
|
|
|
|
|
1. The exec limit is set to 2
2. I'm not sure where to frind the average crawl time
3. Each blog generaly has 1 new post per crawl
4. I have got about 10 feeds assigned per blog
What do you recommend I set the exec limit to?
I will be quite happy with each blog just updating with 1 new post per day, maybe 1 every other day, so how often should I set each cron job to run?
(The server is running fine since my host killed the problem jobs and I changed the cron settings yesterday.)
How many blogs can I use per BO installation?
Generally how many can I put on a server?
Thanks for your help, I am very impressed with the script
 |
|
|
|
|
|
|
|
Atanasis
Owner
Joined: 22 May 2004
Posts: 4284
Location: The Net
|
|
|
|
|
|
|
| Posted: Fri Aug 28, 2009 9:14 am Post subject: |
|
|
|
|
|
|
|
|
|
|
Hm, the exec limits are fine, but i assume there are some dead or slow feeds which make it to run lots of time.
are the blogs BO ones or remote ones?
Generally, there isn't limit about how much blogs to operate, all depends on server and your settings. In my BO i have about 500 blogs and i update them from 1 feed only, once a week. And i don't have the slightest problem about load.
The crawl time, is in your blog settings, right below the list with feeds you assign to blog. Go and see its value for couple of blogs and let me know them..
_________________
Thanks,
Kaktusan
|
|
|
|
|
|
|
|
JohnnyX
Joined: 27 Oct 2008
Posts: 7
|
|
|
|
|
|
|
| Posted: Fri Aug 28, 2009 9:51 pm Post subject: |
|
|
|
|
|
|
|
|
|
|
I understand what you mean now .... all of my blogs are set to a crawl time of 1440
All of the blogs are BO blogs
I have got 690 RSS feeds in the system .... I imagine that some of them must be dead as both you and the tech support at my host have both said that might be the problem .... interestingly the server is still fine at the moment ... perhaps when a blog tries to crawl a dead feed again the problem will reoccur?
Is there an easy way to check for dead feeds?
Also, I think I have used too many feeds per blog, so I am going to start reducing the number ... we live and learn ... |
|
|
|
|
|
|
|
|
|
|
Atanasis
Owner
Joined: 22 May 2004
Posts: 4284
Location: The Net
|
|
|
|
|
|
|
| Posted: Sat Aug 29, 2009 10:28 am Post subject: |
|
|
|
|
|
|
|
|
|
|
yes, dead sites are biggest problems, dunno why.. but it eats lots of CPU when php tries to connect to dead sites. So do the calcs:
560 blogs
1 post a day
10 feed per blog
Thats 560x10 = 5600 feeds downloaded a day + 560 connects to your blogs to publish a post a day.
from those 5600 feeds, lets say 10 to be dead in best scenario, they are making the big load of the borssfeeder.php
Now calc this.. A day has 1440 minutes. Lets say to download 10 feeds for one blog and do the blog update takes more than 1 minute, because some feeds might be slow, other big to download and etc.. It is highly possible all this to take more than 1 minute. So, it might happen BO not to able to update all blogs within a day due to all this.
Generally, once one has more than 200 blogs, he should carefully think how he operates them, because the numbers instantly grow up. You might not have physical time to update them even if its a script..
The thing is that nothing can be done to be fixed, its like that as i said in php tcp connections..
what you may try is:
set the exec limit to 1
edit your cronjobs to be like:
* * * * * /usr/bin/nice -n 15 /usr/bin/php /PATH_TO_BO/borssfeeder.php
put correct path to "nice". That will make the cronjob with lowest priority and limited CPU usage..
Also you may try increasing the posting time to your blogs. There isn't need to update them everyday. If you have high quality handwritten blogs that you have lots of surfers daily and the new content makes you sales for sure, with mass blogging it isn't like that. Posting every 3-4 days to such blogs is pretty much fine. I post once a week. I know people posting twice a month..
_________________
Thanks,
Kaktusan
|
|
|
|
|
|
|
|
JohnnyX
Joined: 27 Oct 2008
Posts: 7
|
|
|
|
|
|
|
| Posted: Sat Aug 29, 2009 2:57 pm Post subject: |
|
|
|
|
|
|
|
|
|
|
| ok, thanks for all your help. I will make some changes. I realise now that I don't need to post once per day so will change that first |
|
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2002 phpBB Group
|
 |
|
 |
| |
