You're running ssh-agent on a desktop (or laptop) system which you leave either suspended or locked (you hope) via a screensaver (see the recent xscreensaver hotkey exploit/bug).
You're running ssh-agent directly via a shell on remote hosts (bad idea).
You can set a timeout for ssh-agent keys with the '-t life' option (default: seconds). Or when adding an identity. However there's no way to specify this in a config file (for sane defaults), and most mechanisms for launching ssh-agent don't allow the user to interact with the initiation in any sane way (e.g.: /etc/X11/Xsession*).
Specifying, say, 43200 - 86400 seconds (for a desktop), or some low multiple of 3600 seconds (for remote sessions) might be reasonably sane.
I'd pick agent forwarding over remote agents myself.
You can list and remove keys with ssh-add:
See the ssh-add man page for more options.