Yes, it is relevant to algebraic geometry, which is important for the treatment of down-to-earth problems in number theory.
I think you’re confusing topos theory with pointless topology. The latter is a fragment of the former and a different fragment is used in algebraic geometry. As I understand it, the main point of pointless topology is to rephrase arguments to avoid the use of the axiom of choice (which is needed to choose points). That is certainly a noble goal and relevant to down-to-earth problems, but not so many in number theory.
I think you’re confusing topos theory with pointless topology. The latter is a fragment of the former and a different fragment is used in algebraic geometry. As I understand it, the main point of pointless topology is to rephrase arguments to avoid the use of the axiom of choice (which is needed to choose points). That is certainly a noble goal and relevant to down-to-earth problems, but not so many in number theory.