Butler means, I think, that it is neither safe nor wise for us to be utilitarians — to hold that virtue consists in actions tending to promote the 'greatest happiness' of mankind.
'Some of great and distinguished merit' (Shaftesbury?), he writes, 'have, I think, expressed themselves in a manner, which may occasion some danger to careless readers, of imagining the whole of virtue to consist in singly aiming, according to the best of their judgement, at promoting the happiness of mankind in the present state; and the whole of vice, in doing what they foresee, or might foresee, is likely to produce an overbalance of unhappiness in it; than which mistakes, none can be conceived more terrible.—Three Sermons p99 by Joseph Butler
Why is the 'greatest happiness' principle (which for good utilitarians comprised the whole of the law and the prophets) a terrible mistake? Because, in Butler's view, we simply do not possess, in our present imperfect state, sufficient clairvoyance for us to be certain about the consequences of our actions,
'nor do we know what we are about, when we endeavour to promote the good of mankind in any ways but those he [God] has directed'.
Shocking actions might be performed with a view to producing an overbalance of happiness. Such ultimate objectives as the happiness of mankind are beyond our limited scope, and we may easily, in attempting to pursue them, become moral monsters. Our safety lies in following the God-given instinct which condemns violence, injustice, and falsehood, and approves benevolence towards some (e.g. kindred, friends, or countrymen) rather than towards others; whereas with the more grandiose aims we may easily be led into perpetrating the former, and neglecting the latter.
'The happiness of the world', he concludes, 'is the concern of him who is the Lord and the proprietor of it';
our concern is only with conduct dictated by the sense of duty. Or as Dr Broad expresses it: God might be conceived to be a utilitarian 'were his moral character merely that of benevolence; yet ours is not so'.
'And though it is of course our duty,' Butler adds, 'within the bounds of veracity and justice, to contribute to the ease, convenience, and even cheerfulness and diversion of our fellow-creatures: yet, from our short view, it is greatly uncertain whether this endeavour will, in particular instances, produce an overbalance of happiness upon the whole; since so many and distant things must come into the account.
Butler has a strong sense (akin to Burke's) of the incalculable complexity of the moral world, and a corresponding distrust of the calculating principle as a moral dynamic. We are so apt to overlook some vital term in making our equations, that our only safe course is to leave moral arithmetic alone, and keep to conscience and the sense of duty.