How to add key value pairs to an array in wp post meta?

I am making a rating system for my site and I would like to store the user id and rating value as key value pairs in array that is stored as post meta.

My problem is that the code I came up with takes the previously stored array, pushes it to a sub-array, and than adds my new key value pair. Then when the next rating comes in, that whole array again becomes a sub array and the new key value pair gets added.

    // Get post meta
    $star_ratings = get_post_meta( $post_id, 'star_ratings', false );

    $star_rating_key = get_current_user_id();
    $star_rating_value = $rating;

    $star_ratings[$star_rating_key] = $star_rating_value;

    // Debug
    debug_to_console( print_r($star_ratings ) );

    // If we fail to update the post meta, respond with -1; otherwise, respond with 1.
    echo false == update_post_meta( $post_id, 'star_ratings', $star_ratings ) ? "-1" : "1";

Here is what I get after the first rating comes in from the debug:

Array
(
    [1] => 5
)

And here is how it looks after the second rating comes in:

Array
(
    [0] => Array
        (
            [1] => 5
        )

    [19] => 3
)

What am I doing wrong or how should i do this?
I would like to have it as:

Array (
[1] => 5,
[19] => 3
)

Or would it be a better way to just create a separate table and store the ratings there?