Does an oil additive that stops oil leaks really work?

After market additives is a no, no. There are varied products out there mostly made up of polymers that tend to grow in molecular size when exposed to temperature. However, the downside is they can disrupt the balance of your engine oil additives and increase oil viscosity at times. It can even lead to oil starvation which can be costly. Suggest you get the high mileage engine oil which already contains a seal conditioner. The effect of reducing the leak is gradual but over time they will not pose a risk of damaging your engine.