Assigning post-specific roles to users in WordPress

I am designing a plugin that need to assign users to a particular post (of a custom type).

For instance, I have a custom post type: ClassifiedFile

For posts of that type, I will need somewhere in the interface to assign users to it, each having different capabilities. In that case, that could be:

• Reviewers assigned to a particular classified file can read it and mark it as approved
• Readers assigned to a particular classified file can only read it
• Or course, readers and reviewers do not have access to other classified files than the ones they are allowed to see.
• Managers who can assign Reviewers and Readers to classified files
• PluginAdmin who can assign Managers to classified files

Ideally, the solution should lend to efficient requests of the type:

• I want to list all classified files a user can read (be it as reviewer or as reader).

So far :

• I have stored a few particular properties in the meta data of the post (such as the approval status).
• I have created custom capabilities : “plugin-admin”, “manage-file-users”, “approve-file” and “read-file”
• I provide two custom roles for that : File Group Manager (“plugin-admin”), File Group Manager (“manage-users”) , File Reviewer (“approve-file”, “read-file”) and File Reader (“read-file”)

I must say I am struggling to find a nice way to address the listing and storing of priviledges per classified file. Ideally I’d like to avoid having to create a separate DB table but if that is the way to do it then I’ll do that.