It's definitely a interesting situation they must giving the cat something they enjoy very much, while I understand you would look after the cat better I would say that for now let the cat decide where it wants to stay, usually the more you force the more the cat will resist, if the cat returns offer it food and drink eventually the cat may return to you or they like having 2 owners for now. Keep us updated.