There are no perfect heating and cooling solutions. You will always have some loss, no matter how efficient the system is, but a forced air system that has leaky ducts can be one of the biggest energy wasters in a home. Ducts are more likely to start leaking near the HVAC unit since the pressure is much higher there, but they can happen along the way in seams or where there has been damage to the ducts as well. The problem is that usually the air from your HVAC unit will leak into an area of the home that is either really hot or really cold and then it will pick that air up along with any odors to channel that into your living space.
Having your ducts inspected by a professional can let you know if you have this issue. Using thermal imaging will really highlight areas where there are temperatures that are outside the norm.