Orthogonal subspace projection (OSP) has been successfully applied in hyperspectral image processing. In order for the OSP to be effective, the number of bands must be no less than that of endmembers to be classified, i.e., the number of equations have to be more than or equal to that of unknowns. This is known as Band Number Constraint. Such constraint is not an issue for hyperspectral images since they generally have hundreds of bands. However, this may not be true for multispectral images where the number of signatures to be classified might be greater than the number of bands such as 3-band SPOT XS images. The generalized version of OSP has been developed, called generalized OSP (GOSP) to relax this constraint in such a manner that the OSP can be extended to multispectral image processing in an unsupervised fashion. The idea of the GOSP is to create a new set of additional bands that are generated nonlinearly from original multispectral bands prior to the OSP classification. Since those additional bands are generated nonlinearly, for linear mixture model, this also introduces error. In this paper, we analyze the error resulting from band generation process with each nonlinear function used for generating additional bands. And then we further propose an approach to select a set of nonlinear functions for GOSP which will yield better classification results.