The earlier treatments of the Lorentz covariant harmonic oscillator have brought to light various difficulties, such as reconciling Lorentz symmetry with the full Fock space, and divergence issues with their functional representations. We present here a full solution avoiding those problems. The complete set of Fock states is obtained, together with the corresponding explicit wavefunctions and their inner product integrals free from any divergence problem and with Lorentz symmetry fully maintained without additional constraints imposed. By a simple choice of the pseudo-unitary representation of the underlying symmetry group, motivated from the perspective of the Minkowski spacetime as a representation for the Lorentz group, we obtain the natural non-unitary Fock space picture commonly considered, although not formulated and presented in the careful details given here. From a direct derivation of the appropriate basis state wavefunctions of the finite-dimensional irreducible representations of the Lorentz symmetry, the relation between the latter and the Fock state wavefunctions is also explicitly shown. Moreover, the full picture, including the states with a non-positive norm, may give a consistent physics picture as a version of Lorentz covariant quantum mechanics. The probability interpretation for the usual von Neumann measurements is not a problem, as all wavefunctions restricted to a definite value for the 'time' variable are just like those of the usual time independent quantum mechanics. A further understanding from a perspective of the dynamics from the symplectic geometry of the phase space is shortly discussed.