The two head coaches golfed together at the NFL's annual owners meetings in March and decided to put a wager on their game.
"We wanted to liven up the match, and we decided to bet and we chose jersey colors," Pederson said Wednesday, according to ESPN's Tim McManus. "Three holes into it, ended up beating him, and that was it."
The Saints will wear their white "color-rush" jerseys Sunday.
Pederson and Payton made another golf bet in July at the American Century Championship in Lake Tahoe, with Pederson coming out on top once again. As a result, Payton had to make a donation to the Eagles Autism Challenge.