The loss was tough on the hosts whose battling performance deserved at least a replay but it was the Scottish League Two side who booked a third-round tie with Tranent at Forthbank on Saturday, November 27.
There were few chances in the opening quarter of the game at a blustery Shielfield but there were promising signs from the hosts midway through the half, creating a couple of openings following good raids forward from Taylor and Stewart.
However, it was Albion who broke the deadlock on 25 minutes when Dylan Mackin’s header went in off the post after McGregor had nodded a Jack Leitch corner into his path.
Rangers refused to buckle and they found the perfect response just eight minutes later when on-loan Hearts midfielder Aidan Denholm smashed a superb shot into the corner from outside the box.
A rasping strike from from Leitch on the turn forced Andy McNeil into action on 38 minutes as the two sides went into the break on level terms.
Both teams created good chances on the resumption, the first falling to Rangers in the early stages, Calum Smith and Graham Taylor combining before Sean Stewart fired over the bar from 20 yards.
Albion then went close, Dylan Bikey creating the opportunity for Kieran Moore but he headed into the side netting with the goal gaping.
Stewart missed a brilliant chance for Berwick midway through the second half when he fired wide when clean through on goal and the hosts went close again on 73 minutes when a long ball through found its way through to McGrath but his shot was saved by Blair Currie.
The home fans would have been wondering if those chances would come back to haunt their team and sure enough it was the visitors who finished the stronger as they pushed for the winner.
Jordan Sinclair headed off the line from a Kyle Banner shot with only five minutes of the game remaining but cruel heartbreak came in the third minute of stoppage time when McGregor pounced on another Leitch corner to grab the dramatic winner for Albion.