Comment 3 for bug 60600

Yeah, I was just coming to the same conclusion. console-setup's FAQ (not yet in Ubuntu) says:

1.1) The font is broken after I switch to X and back to the console

   Actually the boot-time font replaces the font of console-setup. In
   most cases this means the non-ASCII symbols are replaced by some
   other funny symbols.

   This happens because sometimes X video driver is unable to restore
   properly the state of the console when you switch from X to
   text-mode. How severe the problem is depends on the combination
   Video-card/X-driver you use.

   Notice that all this can happen only if the console is in
   text-mode, i.e. you are not using framebuffer. If you add to
   /etc/modules the kernel module for framebuffer you will be able to
   switch from X to console whenever you want and the font will never
   be broken.

   Sometimes the problem doesn't exists at all. It is rumored that
   some old video adapters (more than 10 years old) fall in this
   category but I have never seen such videocard.

   The most common case seams to be that the state at the time the X
   server is started is remembered. Try to setup the console before X
   Window is started and if the problem disappears then you are in
   this category. The problem will reappear if you change the console
   setup while X Window is already started. All video adapters I have
   had are in this category.

   And sometimes the console setup is never remembered. You will have
   to use the "setupcon" utility to restore the console setup and hope
   that the problem will be fixed in some future version of X Window.
   Another option is to use framebuffer. You may want to send a bug
   report to the X developers.