In order to evaluate the effectiveness of product design methods based on Kansei engineering on electronic commence website, a practical and effective AB testing mechanism was proposed. In the mechanism, the target and measures for product design were defined. In experiments, the customers were divided into several groups randomly and the differences were compared between them. Compared with the traditional evaluation method,the customers in the test didn't aware of that they participated in a test. The original online product version and the improved one exist at the same time, and the results are affected by the same external factors. So the evaluation conclusions obtained from the comparison were more accurate and credible. The case study on a commerce APP and hotel booking APP was carried out to prove that the method proposed is effective and feasible. The method has certain referential significance for the Kansei design and design evaluation of electronic commence websites.