Presentation
B-Ring:An Efficient Interleaved Bidirectional Ring All-reduce Algorithm for Gradient Synchronization
DescriptionThe prevailing Ring all-reduce technique in distributed computing comprises communication establishment, data transmission, and data processing phases in each step. However, as nodes increase, it suffers from excessive communication overhead due to underutilized bandwidth during communication establishment and data processing. To address this, we introduce a bidirectional ring all-reduce (B-Ring) approach, employing asynchronous communication to alleviate communication establishment and data processing impact. Extensive experiments demonstrate B-Ring's effectiveness, reducing average communication overhead by 8.4% and up to 23.6%.
Event Type
Work-in-Progress Poster
TimeTuesday, June 256:00pm - 7:00pm PDT
LocationLevel 2 Lobby
AI
Autonomous Systems
Cloud
Design
EDA
Embedded Systems
IP
Security