quit or not

i love my job...
but my co worker want me to quit my job... cause shes not comfortable working with me...
coz we have irregularities...

what should i do