close-pull-requests failing with KeyError/ValueError exceptions

Jeremy, i've a patch for this.

Can you let me know where to get a sample projects.yaml to test the patch?

I've tested with snippet in the script comment, but it's not inline with format parsed in manage_projects.py.

The current state of our production and development projects lists is maintained at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/review.projects.yaml and http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/review-dev.projects.yaml but note that these were very recently split from what is now the jeepyb configuration in the http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/review.projects.ini.erb and http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/review-dev.projects.ini.erb templates.

Thanks!

perhaps. the fix for this bug is mostly inline with changes Monty made recently in manage_projects.py script.
https://github.com/openstack-infra/jeepyb/commit/5986ddef6e0d28eaff99ed42d8ce82d6da16e2f5

Agreed, probably need to do something like manage_projects.py's...

    configs = [config for config in yaml.load_all(open(PROJECTS_YAML))]
    if os.path.exists(PROJECTS_INI):
        projects_yaml_list = configs[0]
        defaults = ConfigParser.ConfigParser()
        defaults.read(PROJECTS_INI)
        [...]
    else:
        projects_yaml_list = configs[1]
        defaults = configs[0][0]
        [...]

Ultimately I think we need to be going through jeepyb.utils.ProjectsYamlRegistry(), maybe indirectly via new functions in jeepyb.projects.

Fix is in progress at https://review.openstack.org/#/c/71110/. Bhuvan, feel free to continue on this bug, we just needed it fixed asap and the change at 71110 isn't how we should fix it long term. Instead we should update all of jeepyb to go through jeepyb.projects to get at the project registry. That is the long term fix. (Note this will probably require updates to jeepyb.projects to support a list of projects and the make ini vs yaml loading selectable).

Clark, i've a patch for jeepyb, fixing jeepyb.utils to use ProjectsRegistry(). The yml/ini loading is done in here. The manage_projects.py, close_pull_requests.py and create_cgitrepos.py are fixed to use u.ProjectsRegistry(). Can you let me know how to test this fix?

I'm unable to setup gerrit in my vm using puppet, as I'm using oraclelinux vm.

Fix proposed to branch: master
Review: https://review.openstack.org/71323

Yep, with jeepyb==8948a8d we're now back to the original failure...

github@review:/$ /usr/local/bin/close-pull-requests
Traceback (most recent call last):
  File "/usr/local/bin/close-pull-requests", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/jeepyb/cmd/close_pull_requests.py", line 93, in main
    org = orgs_dict[project_split[0].lower()]
KeyError: 'openstack-dev'

A quick test confirms that ghub.get_user().get_orgs() is returning an empty list (well, empty PaginatedList custom iterator). I'll have a deeper look and see whether I can spot the cause.

Reviewed: https://review.openstack.org/71323
Committed: https://git.openstack.org/cgit/openstack-infra/jeepyb/commit/?id=f0bec296c63c24beeb9b01c849be2523fcb70be7
Submitter: Jenkins
Branch: master

commit f0bec296c63c24beeb9b01c849be2523fcb70be7
Author: Bhuvan Arumugam <email address hidden>
Date: Wed Feb 5 22:54:13 2014 +0530

    Load project settings from ini file, if present.

    We manage project settings in 2 files, projects.ini and projects.yaml file. If
    ini file is present, read defaults from this file and parse yaml using new
    format.

    Closes-Bug: 1275824
    Change-Id: I5bc2e81388420ce03d5379b8d75063481887243b

Note that after some heavy debugging and experimentation, I can not seem to get the pull-request-closer to see anything but an empty orgs list (testing the same API calls via a Web socket authenticated as the same user also returns an empty array). Not sure what changed with GitHub, but at this point we're probably better off just obtaining the project list from the YAML file/registry and enumerating it that way anyway.

As Clark pointed out in IRC last night, https://developer.github.com/v3/oauth/ says we need the read:org scope on our token. I added it at settings->applications->personal token access and a manual run of the close-pull-requests script succeeded (and spammed us about closing 30-40 pending pull requests--yay!).