exotic characters in the email adress like "ü,ä,ö" are removed

Lots of things have changed since then, there are some Internet Drafts and an
active mailing list on it. Please see http://www.imc.org/ietf-imaa/index.html

William, the mailing list and IDs are about non-ASCII characters in the local
part of email addresses. That's not what IDN is about.

Should this bug be treated as meta-bug for IDN in MailNews?

I propose to split the task of implementing IDN support into three steps with at
least three bugs:

1. Enable the user to send messages to users with IDN in their email addresses.
   - The envelope To should get ACE coded when sending the mail.

2. IDN of email addresses in outgoing messages get ACE coded, mail servers
(SMTP, POP, IMAP) with IDN become usable.
   - Server names and mail addresses in identities should be saved in UTF-8
     (not currently done as I've noted to my surprise).
   - Testing vor valid server names (currently done at least for SMTP servers)
     should be relaxed or done after the server is ACE coded.

3. Email addresses and any other header informations to be displayed should be
presented unencoded as IDN to the user.

Another issue is that MailNews and Thunderbird STRIP non-ASCII characters in the
domain name before contacting the SMTP server.

Here's the output from the mail log when attempting to send to
stale@blåbærsyltetøy.no:

Feb 17 10:57:12 lakepoint sm-mta[2554]: i1H9vCda002552: to=<<email address hidden>
>, delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=30424, relay=blbrsyltety.n
o, dsn=5.1.2, stat=Host unknown (Name server: blbrsyltety.no: host not found)

Oops.

I suggest that until IDN support is present, MailNews/Thunderbird should respond
with an immediate error message to the user when non-ASCII characters are entered.

As it is now, the user will not get an error message in case the stripped name
is a valid domain name ("røed.net" -> "red.net" is a concrete example) and the
local part is valid at that domain.

Christian: IMAA tries to internationalize the entire email address, which
includes IDN on the right hand side of the '@' sign. The effort is currently
focused on the local part because it is quite a pain in the a**.

But since the title of this bug is 'IDN in email addresses', I suppose you are
right that we should focus on just that, ignoring the local part.

Jan: I agree we should warn the user when a non-ASCII address is used until a
proper IDN patch is in.

Need someone familiar with MailNews to lead this...

I've shuffled Christian's points into the following bugs, does this make sense?

1. Agreed, this bug stays as meta-bug for IDN in email addresses.

2. IDN support for mail server hostnames used in account settings. This includes
storage (utf-8 or unichar) and validation as outlined in Christian's (but this
does not deal with IDNs in message handling)

3. Message sending - preparation of 822 headers, envelope sender and recipients.

4. Message display - (Christian's point 3) rendering of IDN within email
addresses (IDNA ToUnicode would be necessary).

I splitted envelope To from the rest (envelope From and 822 headers) because the
former is quite easy to do and would instantly (I've already such a patch)
enable our users to send messages to IDN email addresses.

I don't think we'll have full support until 1.7, but enable sending should IMHO
be available starting with 1.7b (.com and .net domains are already available
with IDN and .de, .at and .info will start next month).

But your split-up makes sense, yes.

Christian: why don't you post your patch?

I have not followed mozilla development for quite some time, don't know if
things still work the same way now..
Is Naoki still actively working on the project?

Created an attachment (id=141810)
send ACE encoded envelope to

It's a working patch to make mail sending work right now.

The encoding part (with find, convert, truncate and append) should be useable
for further work (with one or two modifications). But it might go into an own
helper function to make encoding email addresses more easy.

I think the place where this conversion happens will move as soon as email
addresses and server names are saved as UTF-8.

taking. Christian, do you think it's a realistic goal to make this work before
1.7beta (perhaps 3 weeks from now)? I'm setting the target milestone 1.8alpha
for the now, but we might make it 1.7final (if not 1.7beta)

I think this will work before 1.7b.
I'm about to finish step 2 (see comment #6 for what it includes) and step 3
shouldn't be that hard.
I'll open a new bug for my patch to step 2 later this day to discuss the code I did.

(In reply to comment #2)
> William, the mailing list and IDs are about non-ASCII characters in the local
> part of email addresses. That's not what IDN is about.

Are we sure about this? Is there unanimity of opinions about using IDN on the
righ side of the "@"? Is that what J. Klensin's propsal allows?

http://www.ietf.org/internet-drafts/draft-klensin-emailaddr-i18n-02.txt

As I recall, there was a big discussion about Hofmann vs Klensin proposals at
the IETF meeting last Fall. Does anyone know what was settled there and if IDN
is the agreed upon fromat for the domain part?

Use of IDN on the right side of email addresses seems to be not unquestioned in
that document.
And having different encodings on the left and the right side in email addresses
is something I dislike too. But IDN on the right side works right now and UTF-8
or other 8 bit encodings are unsafe to use.

To this the big part of my changes (make UTF-8 useable internally for mail
addresses and hostnames) are necessary for whatever case.

I vote for using IDN on right side of email addresses now as proposed. If there
will be another standard for encoding the left side, or even the email address
as whole, in the future, we can simply replace the ACE encoding/decoding
function used in my patch by another one (or remove it, if plain UTF-8 will be
the standard).

I agree with Christian, don't see any reason why not. Of course, this will be
considered an experimental feature.

Let me play a devil's advocate. Is there a strong reason to implement this now?
Why should we be creating an experimental e-mail address format and potentially
have these addresses archived and kept for some time even before the standard is
established? Why is there this rush? Unless the standard question is settled one
way or the other, ISP's are not going to allow such maill addresses requiring an
IDN or UTF-8 solution.

As I wrote before, more and more TLD registrars allow IDN domains, and users
start wanting to use them.

As stated, step one is needed anyways and enables our user to send mails to
users with IDN domains in their emails and to connect to incoming and outgoing
servers with IDN names.

To apply ACE coding to domains in mail headers isn't really needed (besides the
fact, 8 bit characters aren't allowed in mail headers), right.

But though haven't found an RFC on IDN in email addresses I didn't doubt ACE
encoding the domain part is the right way because it's a domain.
And please see RFC 3490, 2. "Examples of domain name slots include: [...] the
part of an email address following the at-sign (@) in the From: field of an
email message header".
So I think it's safe enough to use ACE encoding for domainparts of To:, From:,
Cc: and Bcc fields in mail headers.

(In reply to comment #16)
> As I wrote before, more and more TLD registrars allow IDN domains, and users
> start wanting to use them.

For web page URLs, not for e-mail addresses. ISP's should not start mail
services without the standard question settled. Do you know any mail services
now offering e-mail addresses using IDN?

Does IE now support IDN directly? (At may last check, they didn't. A plug-in
exists but no direct support to the best of my knowledge.)
How about Outlook Express or any other mailers? Will Mozilla be the only
mailer(s) supporting IDN mail address?

> For web page URLs, not for e-mail addresses.

Who forbid this?
I know two IDN domains reachable by email. My attached patch is enough to enable
Mozilla to send a message to this address (already tested). Although the mail
headers are illegal (because 8 bit), but works.

> Will Mozilla be the only mailer(s) supporting IDN mail address?

I read about Mutt is IDN enabled, but couldn't find anything concrete.
Maybe Mozilla will the first mailer, and?
I'm not talking about some proprietary extension for Mozilla so mail will only
work between Mozilla users. As far as I can read (see RFC cited) I don't even
make crude assumptions.
And to say it again, it works with the current infrastructure.

(In reply to comment #18)
> > For web page URLs, not for e-mail addresses.
>
> Who forbid this?

All the standards for IDN in web domain names are now in place. It makes sense
to officially support it. The e-mail address part is under discussion now. No
credible mail service will be implementing it. I'm saying that we should be very
careful before we start generating msgs with such a temporary status of the
proposal(s). Given the current state of the use of non-ASCII e-mail addresses, I
see no reason to rush. Perhaps it is not a bad idea to put in some codes now to
eventually deal with it, but I would like to caution against generating e-mail
messages without more progress on the standards for it.

I have to agree with Kat more or less. I shouldn't have been too excited about
this :-). Although using 'punycode' at the right hand side of '@' (the domain
part) would work most of cases (MTAs wouldn't have a trouble dealing with it
because DNS servers would be able to resolve them transparently ), reading the
draft refered to in comment #12 (see section 7. the author is clearly reserved
about using 'punycode' as well as 'utf-8') convinced me that it's premature to
go ahead at this point. Besides, non-Mozilal recipients wouldn't be fond of
their email addresses (domain part)in punycode. Although not in their native
scripts, ASCII-only domain names make some sense, but domain names in punycode
must be very cryptic to them. Well, you can tell them to switch to
thunderbird/mozilla, then :-) However, I guess we shouldn't take that road
(implementing something that's not yet completely agreed upon and using it as
our selling point)

We can still work on it and prepare ourselves 'ready' but even if the patch goes
in, it should go in blocked with '#ifdef 0' (or blocked by a preference entry
disabling it by default). I'm still excited about this possibility and glad that
you came up with a patch. However, working on it is one thing and enabling it
now seems to be another.

Using IDN in email addresses is possible. If someone wants to use them now, one
has to type the mail address e.g.
<email address hidden>.
This will be the envelope and the To: header field.

Using the server internetdomänen.com requires the user to enter
xn--internetdomnen-gib.com in host fields.

Why should we be careful to generate such headers and envelopes automatically
and do what IDNA is about: hiding ACE/puny coded domains from the user.

The changes as outlined in comment #6 won't create non-standard behaviour, not
for now and not for the future.
<email address hidden> is and will stay a valid address, may the
cited draft become a standard or not.

If you don't want to see ACE coded domains in mail headers, you should check a
patch in that makes it impossible for the user to create messages with these
addresses even by hand.

I pondered over the arguments, and read Klensin's draft again.

If Klensin's draft was accepted (a concensus will probably not be reached for a
long time to come), there is probably no need to ACE encode the domain since it
might be UTF-8 (comment #20). However, the MUA obviously has to be modified to
support it, regardless of the outcome from the IMAA group.

In the meantime, there are already a number of mail services and client plugins
(Verisign's iNav: http://www.idnnow.com/).

I am definitely for the standpoint that we start working towards the goal of
supporting IDN in email addresses, keeping an eye on the IMAA development in
parallel. After all, I don't think it is a small job right?

I would also vote for checking in code with with a preference item disabled by
default. This would encourage people to test the feature without having to
compile their own. That's what we did for IDN, _way_ before the IDN RFCs were
published. Also, it is less likely for other changes to break the build if it
was merely commented out by #ifdef's.

By abstracting the encoding/decoding of i15d email addresses into separate
functions, it would be easy to switch the implementation when the standards are
finally out.

I don't see the urgency of getting it out before 1.7b though. If we make it
that's nice. If we don't, we don't.

Today I finished patches to enable on-the-fly convert from/to ACE when sending
(3.) and displaying (4.) to support IDN.

The en-/decoding is done in separate functions, make them to short-circuit in-
and output depending on a pref is not big thing.

But I'm still hanging in the folder/path thing (see bug 235312).
So it's not possible to use IDN in the incoming server if the incoming server
was specified when creating the account (workaround is to create an account with
a ASCII dummy and then rename the server).

Enabling to use UTF-8 for hostname made it necessary to change the parameter
type of GetRealHostName(), GetHostName() and GetEmail() (resp. the type of the
attributs in IDL), ifdef'ing all calls to this functions is a huge job and the
result is confusing and unnecessary. The changes are done once and won't
influence any other code.

Just to let you know, Opera's mail part is IDN enabled since at least 7.23 (the
version I tested).

Ok. I agree that using #ifdef is not such a good idea. Why don't you add a pref.
entry which is off by default?

Requesting blocking1.7 because Mozilla1.7 is the next long-lived milestone
branch. Although some parts of this bug (like support of IDN-mailservers) may
not be so important, sending E-Mails to wrong domains is (see comment #4).

As Mozilla is silently stripping all special characters, it is impossible to
properly (re)encode the e-mail-address at the local mail relay. You can do this
to enable many other mail-clients like Forte Agent.

I certainly don't think we're going to block on adding IDN support to mail. The
sending mail to a modified (stripped of non-ascii chars) domain is a bit
concerning. I've asked mscott to look at that aspect of this bug (it should
probably be filed separately from this if we are going to talk about blocking a
release on it).

*** Bug 241649 has been marked as a duplicate of this bug. ***

only drivers can set (+) blocking flags. you can request (?) them

oh sorry! made a mistake

*** Bug 256395 has been marked as a duplicate of this bug. ***

i would appreciate to add this kind of functionality and it don't raise any
objections.

Resolving/Processing the domainpart in a browser or in an mua should be the
same? Don't you agree? (view from the user)

on the serverside the admins should take responsibility to add the punycode
domainpart to the mailservers and the world would be fine - again :)

Yes, I know, this might be considered spam, but anyhow. More and more clients
support IDN domain names via punycode now and more and more domains appear with
such a name.

Any ideas about when this patch might be landed on trunk?

Nearly one year ago I demonstrated how to enable users to send mails to
addresses like test@müller.de without the need to enter the domain part ACE encoded.

The patch only affects code in the SMTP protocol handler to ACE encode the
envelope. The patch is simple - only drawback would be, that the header of the
sent mail contains the recipients address as UTF-8 string.

It would be nice to hear if this is not wanted or wanted in general. Do we want
to wait for all specs eventually adopted and a full blown solution implemented -
or does that patch have a chance? Yes or no?

mail headers are only allowed to be ASCII, aren't they?

(In reply to comment #35)
> mail headers are only allowed to be ASCII, aren't they?

Yes, because of that I mentioned that the header of the sent mail would contain
the recipients address as UTF-8 string.

(In reply to comment #36)

Sorry for the delay. I definitely want Mozilla mail/TB to support IDNs in email.

> (In reply to comment #35)
> > mail headers are only allowed to be ASCII, aren't they?
>
> Yes, because of that I mentioned that the header of the sent mail would contain
> the recipients address as UTF-8 string.

 I guess that's certainly a blocker. We can't break RFC (2)822/STD 11 in what
send out on the wire. What's the status of bug 235312 and bug 238016? We may
have to land all of them at once (comment #6)

(In reply to comment #37)
> I guess that's certainly a blocker. We can't break RFC (2)822/STD 11 in what
> send out on the wire. What's the status of bug 235312 and bug 238016? We may
> have to land all of them at once (comment #6)

I was nearly done with patches last April. But I discontinued it, since from the
comments here it didn't look like the work is welcome until IDN specs for mail
have been adopted.

*** Bug 307674 has been marked as a duplicate of this bug. ***

> I guess that's certainly a blocker. We can't break RFC (2)822/STD 11 in what
> send out on the wire.
Shouldn't it just be encoded like e.g. subjects are? Like this:
To: =?ISO-8859-1?Q?stale=40bl=E5b=E6rsyltet=F8y=2Eno?=

*** Bug 332259 has been marked as a duplicate of this bug. ***

Copied from bug 332259 comment #1:

IMA overview :
http://www.ietf.org/internet-drafts/draft-klensin-ima-framework-01.txt

SMTP extensions :
http://www.ietf.org/internet-drafts/draft-yao-ima-smtpext-02.txt

Most documents aren't published yet, but you can find back pointers at the end
of these 2. Several of them have already expired though.

https://www1.ietf.org/mailman/listinfo/ima : the mailing list for IMA / IEE. (a resurrected version of the one in comment #1 ??)

Updating the new IETF working group charter in the URL. The new working group's name is EAI - Email Address Internationalization.

*** Bug 386295 has been marked as a duplicate of this bug. ***

*** Bug 387903 has been marked as a duplicate of this bug. ***

So do I understand that in more than 5 (five!) years since opening this bug, this has not been resolved !? This should be complicated to implement (see Bug 387903 for how this should be implemented, Gmail is implementing this too...).

I can't believe this bug is still unresolved.

It's a SERIOUS bug, guys!! There are thousands of such domains now,
and NONE of them can be addressed with Thunderbird.

I've just installed another email program, and I'm sure many more people will do once people or maybe some press people realize that their problems addressing certain domains are deriving from the email client they used!

The Internet standard document for this (RFC 5336/5337) only were published in September 2008, so even though we knew for five years we want something there, we couldn't even know what to implement until 2 months ago. That's not the fault of anyone at Mozilla but of the IETF working group that needed that long to get something defined - and if you read those RFCs (linked at the bottom of the page referred to by this bug's URL), you'll notice that it's not that simple to do.
I can't speak for those doing the actual coding (I'm a mere project manager in the volunteer SeaMonkey team), but I wouldn't expect this to be included in Thunderbird 3 or SeaMonkey 2 - unless someone of the people interested here comes up with a patch. Our developers are currently busy with higher priority issues, but surely would be happy to help anyone who's actively working on this.

Robert Kaiser wrote : "The Internet standard document for this (RFC 5336/5337) only were published in September 2008, so even though we knew for five years we want something there, we couldn't even know what to implement until 2 months ago."

I wonder if there is not a bit of confusion between the domain name (IDN in email) and fully internationalized email addresses (which include le LHS of the email, the local part before the "@").

IDN in email addresses could be supported several years ago I believe (they were certainly by other products).

I think you are both right. The Problem is, that AFAIK using IDN-Domains in SMTP wasn't explicitly specified. None the less it worked fine to enter the domain in puny code. IMHO a simple puny code converter would have done the job for the domain of the email (which is one option suggested by the RFC if I understood right). This is probably the way other clients went.

*** Bug 480560 has been marked as a duplicate of this bug. ***

This is still an issue. In addition the Error message in my case wasn't exactly helpfull. All non-IDN characters were stripped from the error message and thus the address in the mail and the error message didn't match.

Certainly not blocking any 1.9.0.x, since we're not shipping mailnews off
1.9.0. Nor will it block (or be likely to be accepted for) any security release
of any other already-shipping branch, nor will it block Tb3 days before it
ships, so the only flag to twiddle is Tb3.1.

*** Bug 531070 has been marked as a duplicate of this bug. ***

*** Bug 537465 has been marked as a duplicate of this bug. ***

not blocking 3.1 - marking wanted.

*** Bug 545578 has been marked as a duplicate of this bug. ***