pytz includes a hard coded list of time zone names

common_timezones uses data not found in the binary database, so we may not be able to fix this in pytz. Worst case we need to provide a mechanism for distributions that package pytz in this way to keep this list up to date or dynamically generate it from distro specific sources.

I suspected that we wouldn't be able to generate common_timezones (unless they hack tzcode to symlink the obsolete time zones instead of hard linking them). I wonder if the tzdata maintainers would be open to installing a file containing the list of time zones that says which ones are not obsolete?

Petr Machata suggests loading common_timezones from zone.tab at runtime.

This seems like a great starting point for the list. We might want to manually add a few well known names that people expect, such as US/Eastern, but we do that already when generating the static list.

Sounds like a good start. In the long term, it might be worth updating the tzcode programs to produce a file like /usr/share/javazi/ZoneInfoMappings but maybe in a format more amenable to memory mapped use in C programs.

Zones mentioned in zone.tab form a superset of what's currently provided by pytz in common_timezones, because no links are in common_timezones at all. I think it may be actually more correct to include zones that are links, if such zones are in zone.tab: no backlinks are there, and all zones present there are zones belonging to countries (or crown dependencies, semi-autonomous regions, etc., but they do have ISO country code).

(Different wording of the same: common_timezones should contain all zones that the function country_timezones can return. Plus GMT, US/Eastern, et.al.)

Hi!

This is somewhat related to the problem we have with pytz in Google App Engine. Because of the environment constraints and the short-lived runtime instances, initializing pytz takes an unreasonable amount of time in App Engine. Most of this is because pytz/__init__.py checks for 500+ timezone files when it is simply imported (this alone takes between 0.15 and 0.40 seconds in Google's server).

I wish those checkings were lazy, so that we could implement an alternative approach to check the available time zones. The way it is, we need to patch pytz to use in App Engine.

I started a project to provide a pytz version tuned for App Engine via monkeypatch, but realized that it'd be impossible to make it without actually patching __init__. The result is here:

http://code.google.com/p/gae-pytz/

Please let me know if I can help.

Rodrigo,

I'm attaching the patch for what's described in comment #5. I've done some measurements, and the import takes less time than in the upstream package:
$ for i in `seq 1 50`; do PYTHONPATH=. python /home/petr/hle.py; done | python -c 'import sys; print sum (float (i) for i in sys.stdin)'
1.06896305084

With upstream pytz, I get:
$ for i in `seq 1 50`; do PYTHONPATH=. python -c 'import time;x=time.time();import pytz;print time.time()-x'; done | python -c 'import sys; print sum (float (i) for i in sys.stdin)'
3.13412761688

So that does help quite a bit. The code relies on system tzdata, meaning it might also be easier to keep it up to date.

Bah, the first commandline should have been the same as the other. Makes essentially no difference in time.

Petr,
I'm not sure I understand what you're doing there.

Checking all files in the module globals is what I'm trying to avoid. We zip the zoneinfo for appengine, because of limits in number of files. In the end this results in 500+ not raised IOErrors when pytz is imported (which alone takes .15 to .40 seconds). Without it, pytz loading is unnoticeable.

I'll try to make a patch to make the file list lazily loaded (and thus monkeypatchable).

Instead of building a list of pre-defined timezones and then sifting through to discard any that are not on disk (i.e. doing 500+ opens, that's what's in the trunk), I build the list by traversing system hierarchy (i.e. doing a couple opens that don't end up in exceptions). Common zones are loaded from zone.tab. I don't really know what you try to achieve, so this may not help you at all.

I'm attaching a patch that defers the loading of all_timezones and all_timezones_set until the first access. It applies on top of the first patch. This cuts off another ~50% of the import time. I haven't tested that very much though, and I'm afraid whether the lazy collections successfully enough mimic their eager counterparts.

I'll take a look at the lazier init patch now we no longer support Python 2.3 - writing a lazy set seemed impossible before for Python 2.3 - Python 2.6.

Another approach - since we are distributing the compiled zoneinfo database, we can add an extra datafile containing the information we currently generate at run time - the list of all installed timezones, information parsed from zone.tab. If the resource exists, use and trust it. If it does not, fall back to the existing methods (we need the fall back to support installations that make use of the system's compiled zoneinfo database).

This is fixed as much as we can fix it. pytz copes if zones it expects do not exist in the zoneinfo database. It does not notice extra zones added to the zoneinfo database.

Bug 1995864 has been reported and is relevant to this issue: I think Europe/Kyiv is now being set by users since a tzdata update in 22.04 provided it, but pytz (as shipped in 22.04) doesn't know about it. Any suggestions?

Attached debdiff for kinetic SRU.

Attached debdiff for jammy SRU.

This bug was fixed in the package python-tz - 2022.7.1-2

---------------
python-tz (2022.7.1-2) unstable; urgency=medium

  * Team upload.
  * Dynamically determine list of available and common timezones (LP: #207604)
  * Determine IANA (nee Olson) database version dynamically
  * Add autopkgtests to run unittest and own regression tests
  * Update homepage URL
  * Bump Standards-Version to 4.6.2

 -- Benjamin Drung <email address hidden> Tue, 21 Mar 2023 11:21:11 +0100

The fix seems appropriate for the development release - thanks to all involved for helping with this!

SRU review for proposed fixes for Jammy and Kinetic: could we be clear about what broken user experience we are seeking to fix here in Ubuntu Jammy and Kinetic specifically? Bug 1995864 is reported as fixed in Jammy. So what is the broken user story that we are fixing with this update?

Everytime a new timezone is added to tzdata in an update, this timezone cannot be used in pytz with the hardcoded list of timezones. For jammy this applies to America/Ciudad_Juarez (when comparing pytz to tzdata 2023c).

Test case for jammy:

```
import pytz
pytz.timezone("America/Ciudad_Juarez")
```

This should not raise UnknownTimeZoneError.

Hello James, or anyone else affected,

Accepted python-tz into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-tz/2022.2.1-1ubuntu0.22.10.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello James, or anyone else affected,

Accepted python-tz into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-tz/2022.1-1ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted python-tz (2022.1-1ubuntu0.22.04.1) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

dateparser/1.1.0-1 (amd64, armhf, ppc64el)
senlin/1:13.0.0-0ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#python-tz

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted python-tz (2022.2.1-1ubuntu0.22.10.0) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

dateparser/1.1.1-1 (amd64, arm64, armhf, ppc64el)
kopanocore/8.7.0-7.1ubuntu11 (amd64)
senlin/1:14.0.0-0ubuntu1 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#python-tz

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Retrying those autopkgtests solved the failures.

Verified python3-tz 2022.2.1-1ubuntu0.22.10.0 (kinetic) and 2022.1-1ubuntu0.22.04.1 (jammy):

```
$ python3
>>> import pytz
>>> pytz.timezone("America/Ciudad_Juarez")
<DstTzInfo 'America/Ciudad_Juarez' LMT-1 day, 16:54:00 STD>
```

The autopkgtests are successful:

kinetic:

* amd64: https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/amd64/p/python-tz/20230331_193354_bcf83@/log.gz
* arm64: https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/arm64/p/python-tz/20230401_041113_9b51a@/log.gz
* armhf: https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/armhf/p/python-tz/20230331_193408_e22fd@/log.gz
* ppc64el: https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/ppc64el/p/python-tz/20230331_191940_06a59@/log.gz
* s390x: https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/s390x/p/python-tz/20230401_042402_5cfb8@/log.gz

jammy:

* amd64: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/p/python-tz/20230331_225012_10bc3@/log.gz
* arm64: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/arm64/p/python-tz/20230401_022638_8db57@/log.gz
* armhf: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/armhf/p/python-tz/20230331_200820_3a223@/log.gz
* ppc64el: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/ppc64el/p/python-tz/20230331_203522_2cd20@/log.gz
* s390x: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/s390x/p/python-tz/20230401_024737_ac0b3@/log.gz

The verification of the Stable Release Update for python-tz has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-tz - 2022.1-1ubuntu0.22.04.1

---------------
python-tz (2022.1-1ubuntu0.22.04.1) jammy; urgency=medium

  * Dynamically determine list of available and common timezones (LP: #207604)
  * Determine IANA (nee Olson) database version dynamically
  * Add autopkgtests to run unittest and own regression tests

 -- Benjamin Drung <email address hidden> Tue, 21 Mar 2023 12:45:55 +0100

This bug was fixed in the package python-tz - 2022.2.1-1ubuntu0.22.10.0

---------------
python-tz (2022.2.1-1ubuntu0.22.10.0) kinetic; urgency=medium

  * Dynamically determine list of available and common timezones (LP: #207604)
  * Determine IANA (nee Olson) database version dynamically
  * Add autopkgtests to run unittest and own regression tests

 -- Benjamin Drung <email address hidden> Tue, 21 Mar 2023 12:05:43 +0100