I understand your feelings and although is part of the fun as you said, going into a cave is always a matter to worry about a little even being an experienced diver, more or less, and definitely not a proper experience to get into a difficult or narrow cave for first time cave-diving or being unexperienced diver (I don't know how many dives you have or you had when that happened); There are lots of thungs that can go wrong under water as you may know and more into a cave, beginning by visibility: if you go with someone unexperienced and raise the mud and have low visibility it could be a very stressful situation if you don't know the cave previously; that happened to me once and is a bit scary but, as many things under water, you always have to apply this same rule in most of the cases: calm down, don't get nervous and breath slowly.
Also you have to be sure enough about your skills and procedures under water and in the case of getting trapped into a cave, you have to know how to proceed if needed: getting out your jacket or cutting it out, be able to get the bresther out of your mouth and put it back again and, if after being freed you can again put back all your equipment is ok, but if not, you have to be ready to take the last breath and go on a buoyant ascent.
Anyways I hope your future experiences are better than that one because diving is awesome!