I will both agree and disagree to this thought. It is dependent on the context of this statement. I have also seen people restricting their happiness and area of interests stating age as a reason. Though in a way it is a contributing factor, it need not necessarily be the case all the time. I would like to answer this question in two different aspects:
I agree age is just a number because the actual age of a person is based on their thoughts. If they are energetic and enthusiastic with full of positive energy and self confidence, then age is definitely just a number. I know an Ayurveda expert from India. He was very strong and healthy even in the age of 94. He was mentally strong and and enthusiastic. He was the who said me that age is just a number. It is just that there will be some restrictions to out body but our mind doesn't have any age at all. The desire and happiness that our mind experiences is quite common in all the age. So age is purely a body thing and not a mind thing.
There are certain things that mind is capable of doing and body is capable of doing. Age becomes a hindering factor for things that will need body's help to achieving. I strongly believe that every breath we intake is gradually taking us towards the graveyard. There is no doubt about it. If we are going to consider the capabilities of the human body then age definitely matters and it cannot be considered as just a number.