I’ve solved for it making a computer program, but was wondering there was a mathematical equation that you could use to solve for the nth prime?
No, there is no known formula that gives the nth prime, except artificial ones you can write that are basically equivalent to “the nth prime”. But if you only want an approximation, the nth prime is roughly around nlnn (or more precisely, near the number m such that m/lnm=n) by the prime number theorem. In fact, we have the following asymptotic bound on the nth prime pn:
nlnn+n(lnlnn−1)<pn<nlnn+nlnlnn for n≥6
You can sieve within this range if you want the nth prime. [Edit: There are better ideas than a sieve, see the answer by Charles.]
Entirely unrelated: if you want to see formulae that generate a lot of primes (not the nth prime) up to some extent, like the famous f(n)=n2−n+41, look at the Wikipedia article formula for primes, or Mathworld for Prime Formulas.