du -sh /home/*/
This will check the size of all the users home directories.
Note: I would run this as a cron job that checks size ever 10 minutes or so........
A cron job as root would have access to all the user directories and would be able to change permissions on those that are over the limit. Checking every 10 mins or so would allow a user to correct the overage and it would automatically reset itself if the user deleted some files and went below the limit.
man chmod would give you the information you need to reset the directory permissions.