I Have a server in the cloud running Ubuntu 12.10, with 512Mb of RAM just to run a wordpress website (with approx 1000 visits/day).
MySQL was always crashing then I enabled a 4Gb swap to see if can works… but still crashing… and I need to restart the service every time…
Checking the error log from mysql I noticed that InnoDB appears to be in a loop trying to recover something but I think it can’t… can anyone help me?
131009 17:56:57 InnoDB: 5.5.32 started; log sequence number 242183133
131009 17:56:57 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
131009 17:56:57 [Note] - '127.0.0.1' resolves to '127.0.0.1';
131009 17:56:57 [Note] Server socket created on IP: '127.0.0.1'.
131009 17:56:57 [Note] Event Scheduler: Loaded 0 events
131009 17:56:57 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.32-0ubuntu0.12.10.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
131009 17:57:25 [Note] Plugin 'FEDERATED' is disabled.
131009 17:57:25 InnoDB: The InnoDB memory heap is disabled
131009 17:57:25 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131009 17:57:25 InnoDB: Compressed tables use zlib 1.2.7
131009 17:57:25 InnoDB: Using Linux native AIO
131009 17:57:25 InnoDB: Initializing buffer pool, size = 128.0M
131009 17:57:25 InnoDB: Completed initialization of buffer pool
131009 17:57:25 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
131009 17:57:25 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
131009 17:57:26 InnoDB: Waiting for the background threads to start
131009 17:57:27 InnoDB: 5.5.32 started; log sequence number 242183133
131009 17:57:27 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
131009 17:57:27 [Note] - '127.0.0.1' resolves to '127.0.0.1';
131009 17:57:27 [Note] Server socket created on IP: '127.0.0.1'.
131009 17:57:27 [Note] Event Scheduler: Loaded 0 events
131009 17:57:27 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.32-0ubuntu0.12.10.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
131009 17:58:05 [Note] Plugin 'FEDERATED' is disabled.
131009 17:58:05 InnoDB: The InnoDB memory heap is disabled
131009 17:58:05 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131009 17:58:05 InnoDB: Compressed tables use zlib 1.2.7
131009 17:58:05 InnoDB: Using Linux native AIO
131009 17:58:05 InnoDB: Initializing buffer pool, size = 128.0M
131009 17:58:05 InnoDB: Completed initialization of buffer pool
131009 17:58:06 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: The log sequence number in the ib_logfiles!
131009 17:58:06 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
131009 17:58:06 InnoDB: Waiting for the background threads to start
131009 17:58:07 InnoDB: 5.5.32 started; log sequence number 242183143
131009 17:58:07 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
131009 17:58:07 [Note] - '127.0.0.1' resolves to '127.0.0.1';
131009 17:58:07 [Note] Server socket created on IP: '127.0.0.1'.
131009 17:58:07 [Note] Event Scheduler: Loaded 0 events
131009 17:58:07 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.32-0ubuntu0.12.10.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
131009 17:58:33 [Note] Plugin 'FEDERATED' is disabled.
131009 17:58:33 InnoDB: The InnoDB memory heap is disabled
131009 17:58:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131009 17:58:33 InnoDB: Compressed tables use zlib 1.2.7
131009 17:58:33 InnoDB: Using Linux native AIO
131009 17:58:33 InnoDB: Initializing buffer pool, size = 128.0M
131009 17:58:34 InnoDB: Completed initialization of buffer pool
131009 17:58:34 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 242183143
131009 17:58:34 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 242183153
131009 17:58:34 InnoDB: Waiting for the background threads to start
131009 17:58:35 InnoDB: 5.5.32 started; log sequence number 242183153
131009 17:58:35 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
131009 17:58:35 [Note] - '127.0.0.1' resolves to '127.0.0.1';
131009 17:58:35 [Note] Server socket created on IP: '127.0.0.1'.
131009 17:58:35 [Note] Event Scheduler: Loaded 0 events
131009 17:58:35 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.32-0ubuntu0.12.10.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
131009 17:58:44 [Note] Plugin 'FEDERATED' is disabled.
131009 17:58:44 InnoDB: The InnoDB memory heap is disabled
131009 17:58:44 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131009 17:58:44 InnoDB: Compressed tables use zlib 1.2.7
131009 17:58:44 InnoDB: Using Linux native AIO
131009 17:58:45 InnoDB: Initializing buffer pool, size = 128.0M
131009 17:58:45 InnoDB: Completed initialization of buffer pool
131009 17:58:45 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: The log sequence number in the ib_logfiles!
131009 17:58:45 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
131009 17:58:45 InnoDB: Waiting for the background threads to start
131009 17:58:47 [Note] Plugin 'FEDERATED' is disabled.
131009 17:58:47 InnoDB: The InnoDB memory heap is disabled
131009 17:58:47 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131009 17:58:47 InnoDB: Compressed tables use zlib 1.2.7
131009 17:58:47 InnoDB: Using Linux native AIO
131009 17:58:47 InnoDB: Initializing buffer pool, size = 128.0M
131009 17:58:47 InnoDB: Completed initialization of buffer pool
131009 17:58:47 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: The log sequence number in the ib_logfiles!
131009 17:58:47 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
131009 17:58:47 InnoDB: Waiting for the background threads to start
131009 17:58:48 InnoDB: 5.5.32 started; log sequence number 242183153
131009 17:58:48 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
131009 17:58:48 [Note] - '127.0.0.1' resolves to '127.0.0.1';
131009 17:58:48 [Note] Server socket created on IP: '127.0.0.1'.
131009 17:58:48 [Note] Event Scheduler: Loaded 0 events
131009 17:58:48 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.32-0ubuntu0.12.10.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
I found this occurred when MySQL used up too much RAM. This question and this post both helped with a way to test and a solution.
For me, adding this to my
/etc/mysql/my.cnffile fixed the issue.Here’s how I decided how much memory to allocate. Start everything but MySQL and check how much memory is free. Set your InnoDB buffer to 10% of that amount. For me it was 12M.
There’s a couple of things that will help your installation run without any crashes. I’ve had similar issues in the past with WordPress, and it’s likely that you’re either not utilising your InnoDB correctly. From the looks of the log above, as well as details from the post you’ve added a significant Swap, which has likely either done nothing, or made the problem worse…
First thing’s first, your 4GB swap memory is far too high for such a small server. The majority of your processes are likely being used through swap memory. Reduce the swap memory to 512mb or 1gb and troubleshoot from there. Second thing to do is check out the swapiness configuration. If it’s too high then your system will aggressively use swap memory – which wastes cycles, and slows data recovery, if it’s too low then you might as well not have it at all. The default is 60, I recommend starting at 10 and working your way up from there until you find a “sweet spot” for your server. See this Ubuntu article for how to change it on your system:
https://help.ubuntu.com/community/SwapFaq#What_is_swappiness_and_how_do_I_change_it.3F
Your InnoDB buffer is pretty small at 128.0MB. The 4GB swap memory is likely never touched by MySQL (so your swap is probably being used by Apache/PHP if at all). That’s probably why it’s crashing. Increase your InnoDB by at least double, in reality I’d suggest trying to have the full DB in that buffer, but I understand it’s not really possible due to RAM constraints. Try to aim for 1/2 your RAM size, see how it runs compared to before and lower or increase it depending on your results. MySQL Workbench has a ‘Server Status’ tool that telly you your InnoDB Buffer Usage, ideally you want it to run at just under 100% (80-90% in an ideal world) with enough redundant space in case you get any spikes.
A quick note: If you put your entire DB into RAM you’ll need to do more robust & frequent db backups (it’s volatile, and if everything’s in there and the system looses power that’s you screwed).
If these still don’t make any big impacts, then look into more robust cacheing for your WP site. Loading every page on every request is going to be pretty taxing, and could be resolved without too much effort. Check out the Server Stats page on MySQL and look up how many Queries Per Second your site is running, compared to InnoDB reads/writes per second.
I had a similar problem and was able to finally fix it.
Background
I tried a bunch of tutorials that recommended tuning my MySQL installation and Apache configuration with no luck.
Identifying the Problem
It turns out that my site was being subjected to a brute force attack that targeted the WordPress xmlrpc.php file.
In order to check for this, you should view your apache2 access logs and look for suspicious POST requests. In my case, I found a huge number of POST requests to the /xmlrpc.php file coming from the same IP address (about 2 or 3 requests per second).
Solution
In order to fix this, I banned the offending IP address via Iptables:
Ban IP the malicious IP address (replace example with whatever IP address you want to ban) :
To view blocked IP address:
Reference:
http://www.cyberciti.biz/faq/linux-howto-check-ip-blocked-against-iptables/
I had a similar issue, The root cause was memory. My host advised me to do the following steps and it fixed the issue
Open up the MySQL configuration file.
$ sudo nano /etc/mysql/my.cnf
Change the following lines:
max_connections = 50
key_buffer = 25M
max_allowed_packet = 1M
thread_stack = 128K
table_cache = 25