Mod_Rewrite issue – Show WordPress Content vs. Listing Directory

I have a WordPress blog for our church. We have an actual directory called “sermons” and we also have a section on the site called sermons. I need to be able to go to http://ourchurch.org/sermons and get a listing of sermons or http://ourchurch.org/sermons/sermon-title/ and get the sermon.

However, if I go to http://ourchurch.org/sermons/speakername/sermon.mp3 I need it to play or download the mp3. The same would go for pdf files.

Everything is working EXCEPT the sermon listing page. When I go to: http://ourchurch.org/sermons it gives me a directory listing. I need it to allow the WordPress rules to work to direct it to the correct content page.

Here is what I currently have:

# Use PHP5.4 as default
AddHandler application/x-httpd-php54 .php

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

RewriteCond %{REQUEST_FILENAME} -f
RewriteCond %{REQUEST_URI} ^/sermons/(.+)/*.(mp3|pdf)$ [NC]
RewriteRule ^(.*)$ - [L]

</IfModule>

# END WordPress

What am I doing wrong?

Thanks in advance for your help!