In this article I pose a challenge for attempts to ground all reasons in considerations of value. Some believe that all reasons for action are grounded in considerations of value. Some also believe that there are agent-relative restrictions, which provide us with agent-relative reasons against bringing about the best state of affairs, on an impartial ranking of states of affairs. Some would like to hold both of these beliefs. That is, they would like to hold that such agent-relative restrictions are compatible with a teleological theory, one that grounds all reasons for action in considerations of value. This is what I will argue is problematic. I will argue that agent-centered restrictions will not fit into a teleological theory. If the correct moral theory is a teleological one, then there are no agent-relative restrictions. If there are agent-relative restrictions, then teleology is false. My argument challenges a particular project, of showing that all ethical theories are broadly consequentialist. The attraction of this project is that it promises to preserve what is thought to be compelling about consequentialism—its teleology and maximizing—while also preserving elements of commonsense morality—such as agent-relative restrictions—that have typically been thought of as distinctly non-consequentialist in nature. If my argument is correct, then this promise cannot be fulfilled.