Sorting main query by custom taxonomy slug

Is there a way to sort posts by custom taxonomy (shoe_color) ASC|DESC? I would like to do this using query_posts and passing the orderby=shoe_color into it.

Here is the code I’m trying to work out:

function shoe_color_orderby( $wp_query ) {
    global $wpdb;
    return "(
        SELECT GROUP_CONCAT(slug ORDER BY slug ASC)
        FROM $wpdb->term_relationships
        INNER JOIN $wpdb->term_taxonomy USING (term_taxonomy_id)
        INNER JOIN $wpdb->terms USING (term_id)
        WHERE $wpdb->posts.ID = object_id
        AND taxonomy = 'shoe_color'
        GROUP BY object_id
    ) ";
}
add_filter( 'posts_orderby', 'shoe_color_orderby');