I am used to eat together with my family all the time except the time I were in the town alone. I didn't face any trouble like you mentioned but you have your reason also as every family are not the same. I feel good to enjoy food and a good discussion with my family.