Are you using CONFIG_RELOCATABLE=y for your kernels? If so, CONFIG_PHYSICAL_START should not be a factor.
Are you using CONFIG_ RELOCATABLE= y for your kernels? If so, CONFIG_ PHYSICAL_ START should not be a factor.