The peak-to-average power ratio of orthogonal frequency division multiplexing increased the probability of nonlinear signal generation in visible light communication systems. In order to reduce the bit error rate, an improved k-means algorithm of the nonlinear compensation technology was proposed. The transmission characteristics of signals was considered, and the selection of initial points in k-means algorithm was optimized via Dijkstra. The cluster center of the received signal constellation was found, and the traditional k-means algorithm from falling into the local optimal solution was avoided. The simulation results show that this method is more accurate than the traditional algrithm in finding the constellation position of the signal after nonlinear distortion. Hence, it can effectively solve the nonlinear problem of visible light communication and decrease the bit error rate.