In this work we studied two optical models that describe light reflection on thin film with rough surface, light scattering model and effective medium model. The comparison between spectral reflectance measurement and model prediction indicated that effective medium model is better than light scattering model to describe roughness effect. Effective medium model is a highly nonlinear and complex model. In order to study the observability of such a model, some simple nonlinear examples were studied first. The results indicated that linearized observability does not guarantee the observability of the original nonlinear system. Due to the existence of local minimal the guess for the initial state is very important to determine whether the observer will converge. A moving horizon estimation technique was used to extract film thickness and surface roughness from the normal reflectance measurement. The deposition experiment was stopped at various times and the film was characterized offline. The estimated film properties were compared with offline characterization to evaluate the performance of the observer.