The government hopes that at least 85 per cent of homes in the UK will have one of the devices installed by 2024, but the rollout has been hindered by delays.
Households which refuse to have a smart meter installed could see their energy bills rise rapidly as the cost of maintaining traditional meters is likely to increase, Lord Duncan warned.
Duncan told the industrial strategy committee that the maintenance costs would grow as the rollout of smart meters progresses, as it will be more expensive for suppliers to carry out manual readings.
This is because the majority of customers have already made the switch to the new technology.
Energy suppliers have also been set targets that they are required to meet as part of the push towards smart meter usage - although homeowners do not have to accept one.
Duncan said, "Once you get to the point of moving down from 15 per cent to 10 per cent to nine per cent [still using the old meter system] and so forth, the expense of maintaining these relic meters will be very high."
He also warned energy suppliers who failed to reach their rollout targets would feel "the full force of the law".
Around 16.6 million smart meters have been installed so far across the UK, although the rollout has slowed.
Last month, the Government revealed suppliers will now have an extra four years to install a smart meter in every household, but admitted only half would have one installed by the original 2020 deadline.
The estimated costs of the project also increased to £13.5 billion - more than five times the initial £2.5 billion set aside for it.
Unlike traditional gas or electricity meters, smart devices monitor your energy usage in real time by sending up to date readings to your supplier.
However, one in three households are reporting problems with their smart meter, with issues including displays not working property, to devices stopping working, according to uSwitch.
Older smart meters (known as Smets1 models) have also been found to stop working when households switch suppliers, although this issue should be fixed by 2020.