To reveal their compositions, distributions, and provenances of rare earth elements in the sediment from the South Yellow Sea, the data in the surface sediments in the area were studied systematically. The results show that the content of rare earth elements ranged from 77.2 to 261.78 μg/g, with the average value of 166.46 μg/g. The spatial distribution pattern of rare earth element is distinct, showing relative higher value in the muddy area in the central of the South Yellow Sea, where the content is generally more than 180 μg/g, while the lower value area appears as patches close to the coastal regions of Shandong Peninsula, and the content is mostly less than 130 μg/g. The normalized rare earth elements patterns of chondrite and upper continental crust (UCC) show overall pictures of the entire study area, which is similar to the distribution pattern of rare earth elements in Chinese mainland, indicating clear terrigenous imprint of riverine materials, reflecting the intimacy as the main sediment provenance of the South Yellow Sea. Meanwhile, as shown in the correlation analysis of the characteristic parameters of rare earth elements, the composition of rare earth elements in the South Yellow Sea is significantly positively correlated with the grain size of sediments, indicating that the sediment grain size and mineral composition controlled the content and composition of rare earth elements in the study area. Based on the spatial distribution pattern of rare earth elements and those from the main rivers into the South Yellow Sea, we believed that the sediment of South Yellow Sea is mainly from the Chinese large rivers, including Yangtze River, modern Yellow River, and old Yellow River. The northwest part is mainly from the Yellow River materials, and the coastal current of Shandong Peninsula is its main transport power. The material of the old Yellow River mainly deposit in the west of the South Yellow Sea, and the main driving force is the coastal current of Jiangsu Province. The material in the southern part of the South Yellow Sea mainly come from the material entering the sea from the Yangtze River, which is controlled by the Yangtze River diluted water and the coastal current of Fujian and Zhejiang. The east coast of the South Yellow Sea is dominated by terrestrial materials carried by the rivers in the Korean Peninsula, and the main transport power is the Korean coastal current. Taking La/Yb=11.7 as the demarcation line, the entire South Yellow Sea can be divided into two realms. Most of the western materials are from Chinese mainland, while the eastern coastal area from the Korean Peninsula. The boundary between these two realms is close to the middle line of the Yellow Sea trough. Therefore, we confirm that the terrestrial provenance, sediment grain size, and marine current system pattern control the composition and distribution pattern of rare earth elements in the surface sediments of the South Yellow Sea.