Add/remove action on jQuery toggle

I have Ajax setup and I’m trying to toggle a function from being loaded into wp_head. It works on the first toggle function, but the second function doesn’t register.

function av_maintenance_mode_turn_on() {
    if( !wp_verify_nonce( $_GET['nonce'], 'av-maintenance-mode-nonce' ))
    die( 'Go away!');
    if( add_action( 'get_header', 'av_maintenance_mode_active' ) ) {
        echo 'maintenance_mode_on';
    }
    die();
}
add_action( 'wp_ajax_toggle_av_maintenance_on', 'av_maintenance_mode_turn_on' );

This function works ok. The response is seen.

function av_maintenance_mode_turn_off() {
    if( !wp_verify_nonce( $_GET['nonce'], 'av-maintenance-mode-nonce' ))
    die( 'Go away!');
    if( remove_action( 'get_header', 'av_maintenance_mode_active' ) ) {
        echo 'maintenance_mode_off';
    }
    die();
}
add_action( 'wp_ajax_toggle_av_maintenance_off', 'av_maintenance_mode_turn_off' );

There is no response here. I can echo something before the if statement and the response shows up, so this is likely because remove_action is not working.

Here is my JavaScript:

jQuery(document).ready( function($) {
    var widget_inside = $('#av_maintenance_mode_widget .inside');
    var toggle_on = {
        action: 'toggle_av_maintenance_on',
        nonce: av_maintenance_mode_vars.nonce
    };
    var toggle_off = {
        action: 'toggle_av_maintenance_off',
        nonce: av_maintenance_mode_vars.nonce
    };
    $('#av_maintenance_mode').toggle( 
        function() {
            $.get( av_maintenance_mode_vars.ajaxurl, toggle_on, function(response) {
                if( response == 'maintenance_mode_on' ) {
                    $(widget_inside).addClass('maintenance_mode_on');
                }
            });
        }, 
        function() {
            $.get( av_maintenance_mode_vars.ajaxurl, toggle_off, function(response) {
                if( response == 'maintenance_mode_off' ) {
                    $(widget_inside).removeClass('maintenance_mode_on');
                }
            });
        }
    );
});

EDIT

I tried the same with ‘wp_head’ as the tag as well as add/remove filter instead of action, but I’m still not removing the function from the header.