In the "vacuum polarization" divergence, the problem seems to be how to assign the weights on different alternative paths of events. The momentum p of the closed electron-positron loop can be chosen freely in R^4, and there is no natural way to assign weights to R^4 where the sum (or, more precisely, the integral) of weights is 1.
This is analogous to the problem: "Name a random natural number!" Whatever number you choose as a "random" natural number, it is always smaller than almost all natural numbers! It is not random.
That is the reason why the path integral over such a loop "diverges". There is no sensible "functional measure" that we could define as the weight of a path, if one can freely choose p in R^4, and all values should have the same weight.
We will next try to find out what heuristic reasons led Feynman to introduce his precise formula for the electron-positron loop.
It seems that in the formula we let the virtual photon to create a pointlike electron and a pointlike positron in the same exact location x, let one of them to absorb the momentum q carried by the photon, and then calculate some kind of a "dot product" of the electron and positron Fourier-decomposed wave functions. It is not clear to the author of this blog why the result (which is ill-defined, anyway!) should be the right probability amplitude for the annihilation process to happen.
If the virtual electron and positron start from the same point, and they have (large) opposite momenta, how is it possible that they could meet again and annihilate? How can they form a closed loop?
We could have a real positron and a real electron which are disturbed by the virtual photon, and also obtain large opposite momenta. That is, the particles become virtual. The probability of the particles to meet in the future and reset their momenta, so that they become real particles again, is infinitesimal.
How did Feynman arrive at his formula, which for large absolute values of p is essentially 1 / p^2?
Furthermore, the virtual electron and positron in the loop are assumed to be free - there is no virtual photon line between them in the diagram. Thus, there is no pull between the electron and the positron. That makes the annihilation even less probable.
UPDATE August 25, 2018. The Feynman integral formulas come from the perturbation formulas that are derived in standard textbooks. See, for example, Mark Srednicki's book that is freely downloadable from his web page:
https://web.physics.ucsb.edu/~mark/qft.html
The formulas are a result of rather complicated manipulation of the Schrödinger equation, and present the effect of perturbation as an exponential series where only terms that correspond to Feynman graphs have a nonzero contribution.
The terms in the perturbation series do NOT describe in any way "realistic" virtual particles. That is why in the fermion loop, the virtual electron and positron magically find each other to annihilate. They do not behave in any way like realistic particles!
Maybe we could remove divergences altogether if we wrote the perturbation series using more realistic virtual particles?
UPDATE B August 25, 2018. The idea of the perturbation calculation is to let the Dirac field and the electromagnetic field to interact in a controlled and simplified way.
We let the system to develop as free fields except at certain spacetime points (vertexes in the Feynman diagram) where we let the other field to act as the "source" of the field in consideration.
A more realistic physical model might be to distribute the source effect on a large spacetime patch? Or is the effect concentrated in a single spacetime point essentially equivalent? In quantum mechanics, real photons give up their energy in a small space area, whose diameter is roughly the wavelength. What about virtual photons? Can we say that they give up their momentum in a small spacetime area?
A crucial question is if the Feynman diagram model of a virtual pair creation and annihilation is the right way to model the source effect of the electromagnetic field of the virtual photon.
Feynman seems to think that the virtual photon creates the virtual pair at a single point in spacetime. The electron and the positron are modeled with Dirac delta wave packets, which, as we know, contain all values of the momentum p. Feynman calculates the integral of the product of their wave functions over all spacetime points. If we assume that annihilation happens when the particles are at a distance < ε of each other, then the integral, in a sense, describes the probability of annihilation, though it is not normalized and cannot be normalized!
But is that the right way to model the source effect that the electromagnetic field produces in the Dirac field?
Feynman's integral of the product seems to assume that the wave functions of the electron and the positron are independent of each other. Does that make sense? They should have opposite momenta p and -p?
UPDATE August 28, 2018. I am reading chapter 9 of Mark Srednicki's online book. The idea in approximating the effect of a small interaction term, say g φ^3, in the lagrangian, is to manipulate the source J(x) to imitate the effect of adding an interaction term to the lagrangian of the path integral.
Now the reason for the diverging of the Feynman integrals starts to loom: we ignore the backreaction of the interaction on φ! If the source J would get its energy and momentum from outside of the system, the approximation would be at least somewhat realistic. But the interaction in the real world has to get its energy and momentum from the field φ.
Feynman's formulas do take into account some of the backreaction: we require the energy and momentum to be conserved in the vertexes.
Maybe in the vacuum polarization diagram, we should also take into account the fact that the electron and the positron have opposite momenta p and -p, and therefore the probability amplitude for them to meet and annihilate is much smaller than what can be calculated by the standard Feynman propagator formula.
UPDATE B August 28, 2018. If we are calculating planetary orbits using lagrangian mechanics, special care has to be taken to make the approximations to conserve energy and momentum. We cannot introduce small interaction terms that get their energy and momentum from outside the system because that could easily lead to the system being unstable if we integrate over a long period of time.
In Feynman integrals we do not integrate over a long period of time, but we do integrate over an infinite space of momenta. Any approximation error that grows fast enough with the absolute value of the momentum will make the integral to diverge.
The Feynman integrals of the vacuum polarization give very accurate correct predictions if we restrict the momentum by a largish value |p|. That is the expected behavior if the approximation technique has only small errors when the momenta are small. When momenta are large, the approximation technique may have proportionally increasing errors. That would explain why the vacuum polarization integral diverges in a bad way.
It may be that there is no "new physics" to be found at large energies or momenta, but rather the divergences arise from the approximation technique used in Feynman integrals.
Suppose that we were able to show that, with appropriate backreaction, Feynman integral formulas converge and behave in a nice way. Did we then show that the lagrangian of QED is well-behaved? No. We must differentiate between an approximation on the theory and the theory itself. Even if an approximation of a theory is well-behaved, that does not prove that the theory itself is.
Conversely, if we use an approximation technique to calculate the orbits of planets, and the approximation formula diverges under some conditions, that does not prove that newtonian mechanics is in error and we must postulate some "new physics" to correct newtonian mechanics. If an approximation of a theory behaves badly, that does not prove that the theory behaves badly.
How well do Feynman diagrams capture the full path integral?
The "generic" J(x) that is used in Feynman formulas creates a particle whose wave function spreads evenly to all directions.
A crucial question is if we can simulate a complex interaction term like Dirac adjoint(ψ) B ψ in QED with a generic J(x). A layman would think that the electron and the positron in the vacuum polarization diagram would fly to opposite directions.
Another question is how well do the virtual/real particle paths in the Feynman diagrams capture the entire path integral of all possible field configurations. Specifically:
Conjecture 1. If we introduce a small interaction term to the lagrangian of the full path integral, then we can calculate its effect on the integral value by applying it only to the particle paths described in Feynman diagrams.
Much of particle physics is done under the assumption that a general path integral of all possible field configurations can be calculated very accurately by assuming only a few particles and their paths. If we calculate the path of real electrons under an electric field, we often can do that using classical physics. Is there a proof in literature that the full path integral agrees with the classical physics approximation?
We will now concentrate on proving Conjecture 1. If it is true, then we must treat the positron and the electron in the Feynman vacuum polarization diagram as "realistic" particles. They will have opposite momenta and will not annihilate if the momenta are large. That would remove the divergence of the Feynman integral and we would get rid of renormalization.
UPDATE August 31, 2018. The interaction term in the QED lagrangian is not clear:
1. How does the Dirac field generate the static electric field A of an electron or positron?
2. How does field A act on "another" electron and how do we know what electrons are identical?
3. Since the repulsion between electrons is non-local, how can we describe it if we just switch on the interaction on at one spacetime point? Feynman in his 1949 paper switches the interaction on during a short timeslice, but the toy model "phi cubed" in Srednicki's book has the interaction term switched on at a spacetime point.
4. How exactly should we treat the interaction electric field -> Dirac field in a space which is originally empty of electrons and positrons? That is the vacuum polarization problem.
