How to stop the WordPress plugin ‘ajax load more’ from duplicating posts?

I’m displaying 4 posts on my WordPress site home page and have installed the plugin ‘ajax load more’ as a quick and easy way to add a ‘load more posts’ button to my home page. However, when you click the load more button it loads the latest post, as does the home page therefore you have to click it 5 times before you get a post that isn’t already loaded. How do I get it to load the 5th newest post and go from there?

Here’s the Wp_query for my home page:

<?php

        $args = array(
          'posts_per_page'   => 4,
          'offset'      => $offset
        );
        $query = new WP_Query( $args );

      ?>

<div class="container">
<div class="row">
  <div class="col-md-8 blogpost">
    <div class="blogpost2">

      <?php if( $query->have_posts() ) : while( $query->have_posts() ) : $query->the_post(); ?>


      <a href="<?php the_permalink(); ?>"><?php the_post_thumbnail(''); ?></a>
      <div class="colour">
    <h2><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></h2>
        <p><em>
          By <?php the_author(); ?> 
          on <?php echo the_time('l, F jS, Y');?>
          in <?php the_category( ', ' ); ?>.
          <a href="<?php comments_link(); ?>"><?php comments_number(); ?></a>
          <?php the_excerpt(); ?>
        </em></p>
        <center><a id="rdm" class="readmore btn btn-default" href="<?php the_permalink(); ?>" role="button">Read More</a></center>

        </div>

      <?php endwhile; endif; wp_reset_postdata(); ?>
      </div>
    <?php if ( dynamic_sidebar( 'posts' ) ); ?>
      </div>

and here’s my HTML and php used in Ajax Load More:

<?php

        $args = array(
          'posts_per_page'   => 4,
          'offset'=> 4
        );
        $query = new WP_Query( $args );

      ?>

<div class="container">
<div class="row">
  <div class="col-md-8 blogpost">
    <div class="blogpost2">


      <a href="<?php the_permalink(); ?>"><?php the_post_thumbnail(''); ?></a>
      <div class="colour">
    <h2><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></h2>
        <p><em>
          By <?php the_author(); ?> 
          on <?php echo the_time('l, F jS, Y');?>
          in <?php the_category( ', ' ); ?>.
          <a href="<?php comments_link(); ?>"><?php comments_number(); ?></a>
          <?php the_excerpt(); ?>
        </em></p>
        <center><a id="rdm" class="readmore btn btn-default" href="<?php the_permalink(); ?>" role="button">Read More</a></center>

        </div>
      </div>