WordPress sorting by either post meta key or post_date

I’m having a little trouble coming up with the correct SQL for a WordPress query.

What I have are some posts that contain a custom field called ‘date’. I’d like to fetch ALL posts (both those containing this custom field plus those that don’t) and sort them by date AND post_date. date will take precedence over post_date, i.e., if there’s not a date specified, it’ll use the post_date value instead.

What I have so far:

        $query = "
            SELECT wposts.* 
            FROM $wpdb->posts wposts, $wpdb->postmeta wpostmeta
            WHERE wposts.ID = wpostmeta.post_id 
            AND wpostmeta.meta_key = 'date' 
            AND wposts.post_type = 'post' 
            ORDER BY wpostmeta.meta_value, posts.post_date ASC
        ";

Thx