The people who seem to follow God at the beginning were never saved in the first place. The wheat and the tares look the same at first.

Or they were people who accepted the gospel because they were promised health, wealth, and endless blessings. When all that failed to appear they left.

Or, as discussed by Jesus in the parable of the sower, they're the seed that sprung up but were choked by weeds making them unfruitful.