[tex]y''-4y=0[/tex]
has characteristic equation
[tex]r^2-4=0[/tex]
which has roots at [tex]r=\pm2[/tex], giving the characteristic solution
[tex]y_c=C_1e^{2x}+C_2e^{-2x}[/tex]
For the nonhomogeneous part of the ODE, let [tex]y_p=(a_2x^2+a_1x+a_0)\sin2x+(b_2x^2+b_1x+b_0)\cos2x[/tex]. Then
[tex]{y_p}''=(-4b_2x^2+(8a_2-b_1)x+4a_1-4b_0+2b_2)\cos2x+(-4a_2x^2+(-4a_1-8b_2)x-4a_0+2a_2-4b_1)\sin2x[/tex]
Substituting into the ODE gives
[tex](-8b_2x^2+(8a_2-b_1)x+4a_1-8b_0+2b_2)\cos2x+(-8a_2x^2+(-8a_1-8b_2)x-8a_0+2a_2-4b_1)\sin2x=(x^2-3)\sin2x[/tex]
It follows that
[tex]\begin{cases}-8b_2=0\\8a_2-8b_1=0\\4a_1-8b_0+2b_2=0\\-8a_2=1\\-8a_1-8b_2=0\\-8a_0+2a_2-4b_1=-3\end{cases}\implies\begin{cases}a_2=-\dfrac18\\\\a_1=0\\\\a_0=\dfrac{13}{32}\\\\b_2=0\\\\b_1=-\dfrac18\\\\b_0=0\end{cases}[/tex]
which yields the particular solution
[tex]y_p=-\dfrac18x^2\sin2x+\dfrac{13}{32}\sin2x-\dfrac18x\cos2x[/tex]
So the general solution is
[tex]y=y_c+y_p[/tex]
[tex]y=C_1e^{2x}+C_2e^{-2x}-\dfrac18x^2\sin2x+\dfrac{13}{32}\sin2x-\dfrac18x\cos2x[/tex]
