The first and foremost question that comes to my mind when I think about mathematics is: what is a number? Say, what is the number two? Two, is just one of my many thoughts. A thought which has been inculcated in me by my teachers who got it from their own, etc. Note that the number two should not be confused with the arabic numeral 2. A numeral is just a representation of a number, there may be many different representations (e.g. in many languages of the world) but the number is mostly the same.
Now, when I say two apples, I only mean two apples which approximately resemble the shape of an apple or taste like an apple. The weight, volume, colour, texture, taste etc, of these apples might not exactly be the same. For example, a red apple and a green apple are still considered to be two apples even if one is visibly very small in size compared to the another. So, in the case of apples the notion of the number two is not very precise after all. One might argue that the number two here signifies that the apples are physically seperated from one another. Ofcourse then, this begs the question: what is the property of being physically seperated?
So, there might be a variety of physical interpretations of a number based on the situation. Hence, it is impossible to construct mathematics of numbers based on physical characteristics of the objects. Therefore, numbers in mathematics are only treated as a set of objects with some fixed properties, devoid of any physical meaning. Such meaning can be attached to these set of objects as and when we see fit.
The primary notion in mathematics then is not that of a number but that of a set. Set theory has been developed by Cantor in the 19th century and it forms the foundation of all modern mathematics. Sets with relations between them are at the core of most mathematical disciplines.