Let $a,b$ be positive integers.

When $$k = \frac{a^2 + b^2}{ab+1}$$ is an integer, it is a square.

Proof 1: (Ngô Bảo Châu): Rearrange to get $a^2-akb+b^2-k=0$, as a quadratic in $a$ this has two values: $a$ and $kb – a = (b^2-k)/a$. (The second root is determined in two different ways from the expansion $(x-r_1)(x-r_2) = x^2 – (r_1 + r_2)x + r_1 r_2$.)Now suppose we have $a,b$ such that $k$ is an integer but not a square, by the investigation about roots we have that the second root is a nonzero integer since $k,b,a$ are integers and $k \not = b^2$, futhermore it is positive which is easily seen from its defining equation.

WLOG assume $a \ge b$ so that the second root is strictly smaller than $a$. This leads to a decent, replacing $a$ with the second root.

Proof 2(Don Zagier): Apply reduction theory (specifically, Sätze 1 and 2 of Section 13 of my book on quadratic fields) to the quadratic form $x^2 + kxy + y^2$, which is the unique reduced quadratic form in its equivalence class.

Note that Proof 2 is pretty much the same as Proof 1 when written out in explicit detail, but I could not read Zagier’s book because I cannot read German.

I would like to know more approaches to this and other alternative proofs of this result if possible! Thanks in advance.

I would also be interested in related problems (especially easier ones of a similar nature) and texts which cover the reduction theory in English.

**Answer**

This is, IMHO, one of the most popular (and actually the most beautiful) problems in number theory. It is IMO 1988 Problem 6.

You can find it in these links (most of the solutions are the same as the two you remarked, but you may find some more facts about these problems if you see below):

1st (master link)

This is a cool generalization of this problem (you may see this, too):

**Problem.** *Let $a,b$ be positive integers satisfying
$$(ab)^{n-1}+1 \mid a^n +b^n.$$
Then the number $\frac{a^n +b^n}{(ab)^{n-1}+1}$ is a perfect $n^{th}$ power.*

Here are some nice problems **related to this one**:

And, for the sake of completeness about this problem, see Vieta Jumping Method. (IMO 1988 is the best example for Vieta Jumping!)

**Attribution***Source : Link , Question Author : quanta , Answer Author : Cooperation*