WordPress order query posts on popularity of a custom field

I have a WordPress site that allows a user to “like” a post. This adds a custom field to a post with the user ID in it, like so:

add_post_meta($post_id, 'like', $user_id, false);

So a post can end up with a something like a 100 different ‘like’ keys with a user ID.

So I want to return a leader board of most active users based on likes. So I’d want to query that looked at meta_key ‘like’ for the $user_id’s that appear the most. It would look something like this. Obviously this doesn’t work currently.

$args = array(
   'post_type' => 'post',
   'meta_key' => 'likes',
   'post_status'       => 'publish',
   'posts_per_page'    => 20,
   'orderby'           => 'popularity', // This doesn't exist yet, but I wants it!      
   'order' => 'ASC',
   'meta_query' => array(
       array(
           'key' => 'likes',
           'value' => $user_id,
           'compare' => '=',
       )
   )
 );
 $query = new WP_Query($args);

Is this even possible? Or would I be better of getting all posts periodically, adding up the likes and caching the results?

I know I could make a custom field that incremented as a person like it, but I’d rather come up with a query to get results in real time (more robust when deleting a user etc).

Thanks,
Drew