It turns out it is a default settings issue, we shouldn't ship our default config as xfce4-panel.xml but as default.xml.
It turns out it is a default settings issue, we shouldn't ship our default config as xfce4-panel.xml but as default.xml.