WordPress posts sorted by category and year

I’ve been on a long google search trying to find the solution to this… I am creating a cv manager theme using a WordPress install to control content. I have managed to organise all WP posts in categories but would also like to list those posts in year groupings. Currently I have this:

<?php // Categories
         $cvm_cat_args = array('orderby' => 'name', 'order' => 'ASC' ); 
         $categories = get_categories($cvm_cat_args); ?>

<?php foreach($categories as $category) : ?>

<?php // Posts in category
         $cvm_post_args = array( 'showposts' => -1, 'category__in' => array($category->term_id), 'caller_get_posts'=>1, 'order' => 'ASC' );
         $posts = get_posts($cvm_post_args); ?>

<?php if ($posts) : ?>

<h1><?php echo $category->name ?></h1>

<?php foreach($posts as $post) : ?>
<?php setup_postdata($post); ?>

<h3><?php the_title();  ?></h3>
<small><?php the_time(); ?></small>
<?php the_excerpt() ?>

<?php endforeach; ?>

<?php endif; ?>

<?php endforeach; ?>

I’m stuck on how to display the posts in year order within the category so that the output would be something like this:

<h1>Category Name 1</h1>

<h2>2010</h1>

<h3>Post name 1</h3>
<p>Excerpt...</p>

<h3>Post name 2</h3>
<p>Excerpt...</p>


<h2>2009</h1>

<h3>Post name 3</h3>
<p>Excerpt...</p>

<h3>Post name 4</h3>
<p>Excerpt...</p>

<h1>Category Name 2</h2>
etc

Any help would be mightily appreciated.

Thanks!