I also love privacy and I think anyone close to me doesn't trouble me even if they stay with me for a long time. If I were in your place, I would be happy to spend more time with my little brother.