A review has found that the number of people using out-of-hours services has decreased by one third in six years to 5.8 million in 2013/14, according to the National Audit Office (NAO).
As part of this, a survey involving 878 people has revealed one in five people are not aware of the 111 urgent phone service.
The report showed that A&E visits had risen at the same time as a fall in the number of people using out-of-hours care.
Head of NAO, Amyas Morse said: “NHS England has much to do to help secure improvements throughout the system and to increase its oversight.
“It should also work to raise public awareness of how and when patients should contact out-of-hours GP services and needs to be prepared to take the lead in integrating these services effectively with other parts of the urgent care system.”
The national watchdog said that the results of the poll may be due to the fact that they do not trust the service or their lack of awareness of it.
It also said that the figures could reflect the fact that the out-of-hours call service 111 could be dealing with some of the issues of patients.
In a report released at the beginning of July 2014, 17% of people said they thought the services were poor.
President of the College of Emergency Medicine, Dr Cliff Mann told the BBC that there needed to be a remodelling of the way out-of-hours services are planned.
Cliff Mann told the BBC: “Rather than persuade patients to find their way to services, we need to provide a range of services where the lights are on 24/7.
“That is why we are strongly advocating that out-of-hours care should be co-located with A&E services. This isn't a marketing problem to solve but a real life patient care issue.”
Estimates say that the amount spent on the service was £400m last year, which is £75m less than in 2005, once inflation is taken into account.
In spite of this, the 111 service had relieved some of the workload of out-of-hours services, making it difficult to make cost comparisons.
In 2004, GPs were allowed to opt out of providing out-of-hours care and nine in 10 have done so, meaning health bodies were left to find alternate providers.
Patients Association chief executive Katherine Murphy told the BBC: “Unless this issue is addressed, we will continue to see more pressure on an already overstretched A&E system, as patients visit emergency departments."
Health minister Lord Earl Howe said non-emergency phone line NHS 111 was heling to direct patients to the right service and GPs were taking on more of a role in out-of-hours care.