I am creating a WordPress plugin, which I would like to make as customizable as possible for other developers by providing actions and filter hooks in several places.
Can having too many hooks adversely impact the end user web site performance?
WordPress actions, filters and performance
Having unused hooks (calls to do_action, apply_filters, etc), will not significantly decrease performance. Those are function calls with almost no overhead (other than the function call itself and checking for existence of a key in an array).
And providing action and filter hooks is the right thing to do in any code, used in WordPress.