Rewrite ugly URL to clean URL

Im trying to go from:

http://www.example.com/product?level1=value1&level2=value2&level3=value3&level4=value4

To:

http://www.example.com/product/value1/value2/value3/value4

Sometimes not all values are set, so it also needs to work with only 2 values set:

http://www.example.com/product/value1/value2

The http://www.example.com/product/ page uses its own template file.

This is how my code looks like:

function my_rewrite_rules( $wp_rewrite ) {
    $wp_rewrite->rules =
    array_merge( array(
        '^product/([^/]+)/([^/]+)/([^/]+)/([^/]+)/?$' =>
        'index.php?level1=$matches[1]&level2=$matches[2]&level3=$matches[3]&level4=$matches[4]'
    ), $wp_rewrite->rules);
}

add_filter( 'generate_rewrite_rules', 'my_rewrite_rules' );

function my_rewrite_queryvars( $qvars ) {
    $qvars[] = 'level1';
    $qvars[] = 'level2';
    $qvars[] = 'level3';
    $qvars[] = 'level4';
    return $qvars;
}

add_filter( 'query_vars', 'my_rewrite_queryvars' );

It works when im trying to get the query_vars, but it also gives me a 404 page, and not the right template.

What am I doing wrong?