Find three non-constant, pairwise unequal functions f,g,h:R→Rf,g,h:\mathbb R\to \mathbb R…

Here is a proof that no such continuous functions exist. I’ll use juxtaposition to denote function composition. As noted in the comments,
$$h^2=fgh=f^2=ghf=g^2.$$
Let $e=f^4=g^4=h^4$. Note that $fgf=hf=g$ and $gfg=gh=f$, so
$$ef=g^4f=gf^2gf=gfg=f,$$
$$fe=fg^4=fgf^2g=gfg=f.$$
Similarly $eg=g=ge$ and $eh=h=he$. In particular $e^2=ef^4=f^4=e$, so $e$ is idempotent. Consider its image $X=\mathrm{im}\,e\subseteq\mathbb R$, so $e|_X$ is the identity. We have $\mathrm{im}\,f\subseteq X$ since $ef=f$. Moreover $f|_X^{4}=e|_X=\mathrm{id}_X$, so $f|_X$ is a permutation of $X$, and similarly for $g$ and $h$. (aside: it now follows that $f|_X$, $g|_X$ and $h|_X$ generate a quotient of the quaternion group).

Suppose $|X|>1$. Since $e$ is continuous, $X$ is a (possibly infinite) interval. Any continuous permutation of such a set is strictly monotone (either increasing or decreasing). Moreover the only strictly increasing involution of $X$ is the identity. Indeed suppose $\sigma$ is such a function. If $\sigma(x)>x$ then
$$x=\sigma^2(x)>\sigma(x)>x,$$
a contradiction. Similarly we cannot have $\sigma(x)<x$, so $\sigma$ is the identity.

In particular $f|_X$, $g|_X$ and $h|_X$ are strictly monotone. Since $f|_X=g|_Xh|_X$, they can't all be decreasing. wlog suppose $f|_X$ is increasing. Then $f|_X^2$ is increasing. Since $f|_X^4=\mathrm{id}_X$, applying the above result twice gives $f|_X^2=\mathrm{id}_X$ and then $f|_X=\mathrm{id}_X$.

If $|X|=1$, then we also have $f|_X=\mathrm{id}_X$. In either case, for any $x\in\mathbb R$ we have $g(x)\in X$, so
$$h(x)=f(g(x))=g(x),$$
whence $h=g$.