List custom terms with specific value from taxonomy meta field

I am having trouble on making this work but I’m sure the answer is out there. Basically, what I want is to group the list of custom taxonomies I have by their taxonomy custom meta fields. What I have is a custom taxonomy of ‘celebrity’ and I added an extra meta field on that custom taxonomy and name it ‘Sort Order.’ Now I need to group all ‘celebrities’ that have a Sort Order value of ‘A’, then on another column, group celebrities that are in B and so on..

Here’s my code:

<ul>                
            <?php $args = array(
                    'taxonomy' => 'celebrity',
                    'meta_query' => array(
                            'key' => 'sort_order',
                            'compare' => '=',
                            'value' => 'A',
                             )              
                    ); ?>
            <?php $tax_menu_items = get_terms( $args );
            foreach ( $tax_menu_items as $tax_menu_item ):?>
            <li>
                <a href="<?php echo get_term_link($tax_menu_item,$tax_menu_item->taxonomy); ?>">
                    <?php echo $tax_menu_item->name; ?>
                </a>
            </li>
            <?php endforeach; ?>
        </ul>

Could some please help me make sense of how can I achieve this? Thanks in advance.