Exclude posts which have any term in a certain taxonomy from the query

I have a custom taxonomy, and for a certain query I am doing, I want to retrieve all posts which do not have any terms of that taxonomy assigned (for example, they are tag-less).

At the beginning I thought to do the following:

$tax_slugs = wp_list_pluck( get_terms('my_tax'), 'slug' );
$query_args['tax_query'] = array(
    'taxonomy' => 'my_tax',
    'terms'    => $tax_slugs,
    'field'    => 'slug',
    'operator' => 'NOT IN',
);

However, what this does is looking for posts with my_tax taxonomy, excluding all of them, which is obviously not ideal.

The only other solution I’ve thought of is by merging two queries, but I’d rather keep the number of queries at a minimum, since this is already a secondary query.