WordPress Wp_User_Query order by custom value

I have a custom field in my users field, called order. It’s an input field where I can add numbers that indicate the order in which my users should appear in the query. I also have a custom field that indicates a level of a user (Worker, Support, Outside help…)

I have arguments for the query set up like:

    $args = array(
        'meta_query' => array(
            0 => array(
                'key'     => 'user_select',
                'value'   => 'worker',
            ),
            1 => array(
                'key'     => 'order',
            )
        ),
        'orderby' => 'order'
    );

    $user_query = new WP_User_Query( $args );

but this is not working. The worker one is working (I get users assigned only to workers value), but the order is not working. How can I make it so that the query will output users according to numbers (in ascending order).