How to sort posts by the average of comment meta values

Here’s the situation-

I’ve customized the comment submission whereby users can provide a rating, and this gets stored as a meta value in the commentmeta table. I can query to return the average value and display in the post. No problem so far.

What I want to do now is sort posts by this average rating.

My code to get the average rating is:

function average_rating() {
global $wpdb;
$post_id = get_the_ID();
$ratings = $wpdb->get_results("

    SELECT $wpdb->commentmeta.meta_value
    FROM $wpdb->commentmeta
    INNER JOIN $wpdb->comments on $wpdb->comments.comment_id=$wpdb->commentmeta.comment_id
    WHERE $wpdb->commentmeta.meta_key='rating' 
    AND $wpdb->comments.comment_post_id=$post_id 
    AND $wpdb->comments.comment_approved =1

    ");
$counter = 0;
$average_rating = 0;    
if ($ratings) {
    foreach ($ratings as $rating) {
        $average_rating = $average_rating + $rating->meta_value;
        $counter++;
    } 
    //round the average to the nearast 1/2 point
    return (round(($average_rating/$counter)*2,0)/2);  
} else {
    //no ratings
    return 'no rating';
}
}

I’m thinking I need to start storing the average value as a post meta field otherwise I’d have to loop through queries within queries with multiple joins, which seems to be a really bad idea. I’m hoping someone with a fresh set of eyes can help me think through architecting this.