We’re running a membership site using WP 4.1 and ~ten reputable plugins. ON average, our first request take 1.2 seconds.
Since it’s a membership we use fragment caching, instead of full page caching. Doing this we’ve reduced the MySQL query time from 90ms to 10ms.
Our plugins and theme, while probably not impressive php, is not poorly coded. These long TTFB take place with the default theme and no plugins as well.
What are the remaining factors in TTFB and how can we pinpoint the problem?
Well, if you think you have optimized your code, the next items on your list should be related to server config. Things like software (NGINX instead of Apache http://www.theorganicagency.com/apache-vs-nginx-performance-comparison/ ), adding server cache (Varnish, Redis, Memcache, OPCache, etcetera), MySQL configuration and a lot of other small improvements.
Also, being on a shared hosting environment can be considered as a bottleneck.