Basic frequency is an important index for safety assessment of brick masonry walls. To effectively reduce the interference of noise in basic frequency identification for the walls, a multi-channel dynamic testing scheme was designed, and the Kalman filtering (KF) was combined with the singular value decomposition (SVD) to denoise. First, different damage states on brick masonry walls were imposed by pseudo-static tests, and then the acceleration responses at different parts of the damaged walls were recorded under ambient excitations. As the next step, the KF method that introduced the walls’ physical parameters was used to denoise for each of the acceleration record. The walls’ basic frequency was identified by using each denoised acceleration record separately. Finally, all the denoised acceleration records were combined by SVD, and the walls’ basic frequency was identified again. The identified basic frequencies decrease along with the increase of damage severity. The identification results quantitatively relate the change of basic frequency to the decrease of mortar strength, the increase of the opening rate, and the increase of the aspect ratio, and they quantitatively reveal the damage process of the walls. The research shows that the KF can use reasonable prior judgment about the walls’ physical parameters to effectively denoise and then accurately identify the walls’ basic frequency. The identification results can be further improved by combining useful information in multiple acceleration records through SVD.