[6.0/trunk] Encoding trouble in mail_message parsing and base_action_rule processing

Hello Joël Grand-Guillaume ,

I have checked this issue with trunk and I face the
problem of First Part but Second part of bug is not reproduce
at my end so would you please elaborate more regarding
second part of this issue with video or proper steps.

Thanks and waiting for reply.

Hello Joël Grand-Guillaume @ CampToCamp,

Thanks for this bug report.

We have fixed the First part of the bug report in 6.0,
https://code.launchpad.net/~openerp-dev/openobject-addons/6.0-opw-382010-rha
Revision ID: <email address hidden>
Revision 5031

More over the Second part is not related to 6.0 and should be fixed in Trunk by R&D Team.

Best regards.

"but Second part of bug is not reproduce
at my end so would you please elaborate more regarding
second part of this issue with video or proper steps."

Hello,

To reproduce, you need to send a mail which contains broken chars in its body to a mail you fetch in OpenERP... But
During the fetch of the e-mails, the mail body makes Postgresql issue an "invalid byte sequence" UTF-8 error because it tries to write some not valid UTF-8 chars.

We cannot guarantee the quality of the data which enters in OpenERP via emails, mails may contains invalid characters. But we have to handle this exceptions so we should ignore them or replace them by the replacement caracter �.

Thanks
Guewen

Hi there,

First thank you to review this one. I came back to you as I just face it again on the last trunk version. PLEASE, FIX ALSO THE SECOND PART IN BOTH 6.1 AND 6.0 :

Second part: Fetching mail when some character are broken in the body_html part. It could happend that you receive an email with broking characters in it, you should not block everything because of that. Currently, if it happend, no other mail are fetched, and you got an PostgreSQL error when trying to write it in DB : invalid byte sequence for encoding "UTF8": 0xe96ce9

=> We really need this fix here. I just tested my patch and it also work on v6.1 (first part as well as second one).

To reproduce, you'll need to send a mail to the fecthmail of openerp with a broken char in it like in this script:

# Import smtplib for the actual sending function
import smtplib

# Import the email modules we'll need
from email.mime.text import MIMEText

# Create a text/plain message
msg = MIMEText('abcdef' + chr(255))

# me == the sender's email address
# you == the recipient's email address
msg['Subject'] = 'The broken mail'
msg['From'] = 'YOUR_EMAIL'
msg['To'] = 'TO_FETCHMAIL_OPENERP'

# Send the message via our own SMTP server, but don't include the
# envelope header.
s = smtplib.SMTP('YOUR_SMTP_SERVER')
s.sendmail('YOUR_EMAIL', ['TO_FETCHMAIL_OPENERP'], msg.as_string())
s.quit()

Don't forget to replace those variable : YOUR_EMAIL, TO_FETCHMAIL_OPENERP, YOUR_SMTP_SERVER.

Or just type this in a Python console to have the proof:

a = 'abcdef' + chr(255)
a
>>>> 'abcdef\xff'
str(a)
>>>'abcdef\xff'
unicode(a)
>>>>Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 6: ordinal not in range(128)

You see that unicode can't make it. This is exactly what happen when PostgreSQL try to record the mail, and this stop everything !

Thanks for your review,

Best regards,

Joël

Hello,

In fact fix is not applicable for second part in 6.0 because mail_gateway module became mail and was nicely refactored.

However, receiving mails with special charactere (might be carriage return) can not be unicoded in 6.0. So this issue is also present in v6.0.

I think it is the job of mail_gateway to ensure the encoding of the mail.
Otherwise everywhere else in code we would have to duplicate the same check.

Cheers,
Yannick

Hello Joël Grand-Guillaume ,

I am confirming this issue for first part of bug.
For second part I am not able to reproduce so would
you please provide proper steps with new bug
report.

Thank you!

Hello, Thanks For Reporting.

according to comment #7
It has been Fixed in https://code.launchpad.net/~openerp-dev/openobject-addons/trunk-bug-921442-kjo
revision-id: <email address hidden>
revno: 6522
It will be available in trunk soon.

Dear OpenERP Team,

I understand this is a boring one to reproduce. But trust me, this really can't stay here for the 6.1 release ! If a mail come with an encoding trouble (and it will !), all your mail fetching will be stopped => You'll have no mail received anymore !!!

I can't just let you close this one cause you can't reproduce it. Reading the code is enough to understand there is a trouble.

For this reason, I re-open it. Please consider this part of the patch I just re-post.

Thank you for your consideration.

Regards,

Joël

Could you please submit a proper merge proposal with a test case and we could merge it.

Attach the video for the test case

Tolerance to incorrectly encoded emails is a wishlist. A PYTHON testcase must be written to validate the fix. Only a merge proposal with the test and the fix will be merged.

Hello,

Their are two points proposed in this bug as per Comment #7 by Jignesh Rathod.

So We have tested the first part of the bug which is "Parsing the rules (base_action_rule) breaks if the regexp or the name of the resource (model) has some non-string char". And Technically it is valid bug, we have reproduced the bug and I am adding step here for the same :
Test case :
- Install Module crm.
- Create 'Automated Action' under Settings/Customization/Automated Actions/Automated Actions for the model crm.lead.
   - Now Supply non-character (e.g. "ààààààà") string under following two field :
   - First field Under Conditions on Model Fields Section, 'Regex on Resource Name '
   - Second field Under Section 'Condition on Communication History' Regular Expression on Case History
- Now, When Automated action with non-character string will be trigger we will have traceback on server saying :
 File "/home/jam/rdtools/addons/trunk-calendar-phase2/base_action_rule/base_action_rule.py", line 372, in do_check
    ptrn = re.compile(str(reg_name))
    UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-6: ordinal not in range(128)
    2012-02-15 05:55:44,278 4176 ERROR ? openerp.netsvc: ascii
    ààààààà
   0
   7
    ordinal not in range(128)

 (You can see the Video on Comment#13 for the same)

SO the Merge Proposal Given by Kuldeep Joshi(OpenERP) at Comment#8, Fixes the First Part of Bug,
MP Link : https://code.launchpad.net/~openerp-dev/openobject-addons/trunk-bug-921442-kjo/+merge/92248.

About second Part we do not any case which produce bug.
So I request Community Team to take Appropriate Decision for the Second Part of the Bug.
Update me If I am wrong.

Thank You.

Hello,

@Jigar: Thanks for your unnecessary explanation..lol..;-).

First of all Jignesh is already give a decision about it as well as Antony also. So here would you please consider only 1st part of the bug.

For 2nd part:
@Joel:
I have seen your patch as well as we try some cases to reproduce it but we can't produce it, As per tde says it's wishlist. So I am requesting you either you have to post a new bug report for this with proper test case or you can provide a directly MP for this with test case.

So we can check your this issue properly.

Thanks for understanding!

Dear OpenERP Team,

I can understand you need merge proposal and test case to handle your stuff. Sometimes, it is not that easy to write a proper one in Yaml, especially here. Can I just remind you gently that you never write one test for this module as well ?! This is just because we can't actualy do it, or we will just spend weeks on it. Code a mail sending test and I will include my test case in it I promise !

I try a last time to convince you here by proposing a patch and instructions to reproduce the bug :

1 ) Install a fresh OpenERP DB

2) Install and configure the CRM + mail interface

3) Setup an Incoming mail server on any test mail account you have (let's say <email address hidden>)

4) Send an email @ <email address hidden>

5) Apply the "reproduce.patch" on the server and restart it

6) Ask to fetch the email through : Settings -> Configuration -> Email -> Incoming Mail Server by clicking the "Fetch Now" button

You'll have an error raised by Postgres cause of UTF-8 decoding trouble (here a copy of postgres logs:
ERROR: invalid byte sequence for encoding "UTF8": 0xff
ERROR: current transaction is aborted, commands ignored until end of transaction block
)

Now you can retry the same procedure with my second patch "reproduce_fixed.patch" (that include my fix) and you'll see you're not gonna have this crash anymore.

Now, I take the time to discuss the trouble, I even provide a patch on a feature that is quite important and crucial (even for your own infrastructure, as you also use the CRM for OPW support case).

You now have the possibility to reproduce it... Take your own decision. I have taken my own and committed my patch in our Camptocamp's branch. You can merge it if you want, but I won't request a merge myself for that case :

bzr merge -c6589 https://code.launchpad.net/~c2c/openobject-addons/6.1-c2c-official

Regards,

Joël

Hello ,

@Joel
First of all, Thanks for nice explanation.
According to your comment #17's producible steps, I have reproduce the same at my end.

I have applied your patch and then try again, It's working fine after applying your patch.
Thanks for your contribution and passion for this issue!

@RD team :

I have attached a video which described the bug, So would you please check it and also fix this with given patch or more suitable solution(If possible).

So now I am confirming this issue for the remaining part.

Thanks for understanding!

Hello,

Thanks for the fast review and consideration ! Please, don't forget to patch this issue as well on v6.0 cause it's also there. I haven't provided reproducing step on this one ;), but same trouble is here....

Keep going this way !

Kind regards,

Joël

Hello Joël Grand-Guillaume,

  It has been fix in https://code.launchpad.net/~openerp-dev/openobject-addons/trunk-bug-921442-nco
branch.
  Revision ID: <email address hidden>

  Revision num : 6627.

Thanks,
 NCO.

The fix for the first part of the bug report was merged in 6.0 and 6.1 at revision [1][2], updating status accordingly.

Concerning the second part of the bug report that is mostly unrelated, it is still not reproducible. All the described methods to reproduce do not seem to produce errors, except when directly injecting the error in the code of OpenERP itself, which does not prove it can happen with a normal e-mail. As a result, I'm afraid the proposed patch will not fix what it claims.
In addition, that patch is wrong for several reasons, discussed in the review of the [3] branch. Encoding issues are sometimes complex and if you jump too quickly to conclusions you risk breaking something else that is more important than the corner case you are trying to correct.

Perhaps the best way to make sure we properly fix that second part is to send us the source of a real email file that triggers the problem, as you seem to have seen one. You can safely anonymize the content of the email you manage to do it. Otherwise you can e-mail it to me (as an attachment) and I will see how we can make a proper patch. It might also make sense to handle this in a separate bug report.

For further discussion on the actual patch, do not hesitate to comment on [3].

Thanks,

[1] 6.0 rev. 5075 rev-id: <email address hidden>
[2] 6.1 rev. 6679 rev-id: <email address hidden>
[3] lp:~openerp-dev/openobject-addons/trunk-bug-921442-nco

i face the same proble on OpenERP v7 , is there anyone has solution for V7 ?