If you take quests to improve your relations with a village which belongs to your vassal, depending on the vassal you will take a relatonship hit. When a lord gets taken captive during a war, you will take a relationship hit. When assigning fiefs you will take relations hit to all your vassals but the ones who are friendly to the vasssal you gave the fief to. It took me a few games to figure this out as it isn't outlined in the wiki.
