The downpour led to an early lunch with the Proteas placed at 55-2 while chasing a target of 369. They require 314 more runs with eight wickets in hand.
Quinton de Kock, promoted up the batting order, was batting on 28 with first-innings centurion and skipper Hashim Amla on four at the break.
The hosts, who posted 421 in their first essay, had bowled out South Africa for 282 in their first innings to gain a 139-run lead.
Sri Lanka declared their second innings on 229-8.Perera (1-27) gave the side an early breakthrough when Elgar failed to judge the ball that pitched on the middle but turned to clip the off-stump.
Elgar made 13 after facing 65 balls and shared 43 runs for the second wicket with De Kock.
Elgar's opening partner Alviro Petersen was out for a duck late Sunday, being caught by Kithruwan Vithanage off left-arm spinner Rangana Herath (1-7).
South Africa lead the two-Test series 1-0 after winning the opener in Galle by 153 runs. Even if they draw this game, they will reclaim the number one world ranking from Australia.