Memcached request loop or cache stampede in WordPress

I have a clustered WordPress webapp that is using a Memcached cluster consisting of six nodes to store WordPress’s object cache, but I’m seem to be experiencing some sort of sporadic Memcached set/replace loop or cache stampede that occasionally causes a single node in the Memcached cluster to completely saturate the Memcached and database network link.

The set/replace loop or stampede seems to be caused by lots of attempts trying to “add” the WordPress “wp_:options:alloptions” key which Memcached replies with NOT_STORED to all those particular queries. While at the same time majority of MySQL queries look like this:

SELECT option_name, option_value 
FROM wp_options 
WHERE autoload = 'yes'

The only change that I’ve made to the Memcached Object Cache plugin (http://wordpress.org/extend/plugins/memcached/), is the default expiration setting.

#var $default_expiration = 0; //original
var $default_expiration = 1800;

So far I have been able to mitigate the problem by increasing the Memcached daemon maximum connection limit. I also changed the database storage engines from MyISAM to InnoDB, and also increased the APC shm_size from 128M to 1536M.

Is the object cache set/replace loop or stampede inevitable or is it a consequence of my current setup?

Environment:

• WordPress Version 3.7.1
• Memcached 2.0.2 Plugin http://wordpress.org/extend/plugins/memcached/

• Memcached 1.4.14-1

  PORT="11211"
  USER="nobody"
  MAXCONN="10240"
  CACHESIZE="4096"
  OPTIONS=""
  

• PHP 5.3.28

• memcache Version 3.0.8
  memcache support => enabled
  memcache.allow_failover => 0 => 0
  memcache.chunk_size => 32768 => 32768
  memcache.compress_threshold => 20000 => 20000
  memcache.default_port => 11211 => 11211
  memcache.hash_function => fnv => fnv
  memcache.hash_strategy => consistent => consistent
  memcache.lock_timeout => 15 => 15
  memcache.max_failover_attempts => 20 => 20
  memcache.protocol => ascii => ascii
  memcache.redundancy => 1 => 1
  memcache.session_redundancy => 2 => 2
  Registered save handlers => files user memcache

• APC Version => 3.1.9
  APC Debugging => Disabled
  MMAP Support => Enabled
  MMAP File Mask =>
  Locking type => pthread mutex Locks
  Serialization Support => broken
  Directive => Local Value => Master Value
  apc.cache_by_default => On => On
  apc.canonicalize => On => On
  apc.coredump_unmap => Off => Off
  apc.enable_cli => Off => Off
  apc.enabled => On => On
  apc.file_md5 => Off => Off
  apc.file_update_protection => 2 => 2
  apc.filters => no value => no value
  apc.gc_ttl => 3600 => 3600
  apc.include_once_override => Off => Off
  apc.lazy_classes => Off => Off
  apc.lazy_functions => Off => Off
  apc.max_file_size => 1M => 1M
  apc.mmap_file_mask => no value => no value
  apc.num_files_hint => 1000 => 1000
  apc.preload_path => no value => no value
  apc.report_autofilter => Off => Off
  apc.rfc1867 => Off => Off
  apc.rfc1867_freq => 0 => 0
  apc.rfc1867_name => APC_UPLOAD_PROGRESS => APC_UPLOAD_PROGRESS
  apc.rfc1867_prefix => upload_ => upload_
  apc.rfc1867_ttl => 3600 => 3600
  apc.serializer => default => default
  apc.shm_segments => 1 => 1
  apc.shm_size => 1536M => 1536M
  apc.slam_defense => On => On
  apc.stat => On => On
  apc.stat_ctime => Off => Off
  apc.ttl => 0 => 0
  apc.use_request_time => On => On
  apc.user_entries_hint => 4096 => 4096
  apc.user_ttl => 0 => 0
  apc.write_lock => On => On