A problem from A.P. French's Special Relativity:

My attempt to the first question:

In laboratory frame $S$:

In a zero-momentum frame $S'$ moving with velocity $v$ relative to $S$:

If we can resolve $\gamma(v)$ and $v_3'$, then we can take the derivative of $\tan\theta$ with respect to $\theta'$ and obtain its maximum value.

But I am facing two difficulties:

(1) How to obtain $\gamma(v)$? My attempt:

$$ v_2'=-v \tag{11} $$

$$ v_1'=v_1-v \tag{12} $$

$$ \gamma(v_1)=M/m \tag{13} \quad \text{(given in the question)} $$

$$ \gamma(v_2')=\gamma(-v)=\gamma(v) \tag{14} $$

Since $S'$ is a zero-momentum frame,

$$ \gamma(v_1')Mv_1'=\gamma(v_2')mv_2' \tag{15} $$

Substitute (11) - (14) into (15),

$$ \gamma(v_1-v)M(v_1-v)=-\gamma(v)mv \tag{16} $$

Now we can solve for $v$ and then calculate $\gamma(v)$, but the calculation is very complicated.

(2) How to obtain $v_3'$? Apply energy-momentum relationship on mass $M$ in $S'$ after collision:

$$ (E_3')^2-(Mc^2)^2=(cp_3')^2 \tag{17} $$

Now we have equation (7) and (17) for the three variables $E_3'$, $p_3'$ and $v_3'$. We need one more relationship to solve for $v_3'$.

Besides, it will take too much effort to solve the above equations. I guess there should be a simpler approach. Do you have any idea?

Note 1 : The author only briefly mentioned 4-vectors, so we are not supposed to use it to solve this problem.

Note 2: Another approach is as follows:

$$ \mathbf{p}_1=\mathbf{p}_3+\mathbf{p}_4 \tag{18} $$

$$ E_1+mc^2=E_3+E_4 \tag{19} $$

$$ E_1=\gamma(v_1)Mc^2 \tag{20} $$

$$ E_1^2-(Mc^2)^2=c^2p_1^2 \tag{21} $$

$$ E_3^2-(Mc^2)^2=c^2p_3^2 \tag{22} $$

$$ E_4^2-(mc^2)^2=c^2p_4^2 \tag{23} $$

By (18),

$$ (\mathbf{p}_1-\mathbf{p}_3)\cdot(\mathbf{p}_1-\mathbf{p}_3)=\mathbf{p}_4\cdot\mathbf{p}_4 \tag{24} $$

$$ p_1^2-2p_1p_3\cos\theta+p_3^2=p_4^2 \tag{25} $$

Substitute (13) and (19) - (23) into (25), take derivative of $\cos\theta$ with respect to $p_3$, set it to zero, then we can find the value of $p_3$ that gives the maximum value of $\theta$. But the computation is very complicated.