diff -Nru python-sh-1.12.14/CHANGELOG.md python-sh-1.14.1/CHANGELOG.md --- python-sh-1.12.14/CHANGELOG.md 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/CHANGELOG.md 2020-10-24 17:34:57.000000000 +0000 @@ -1,4 +1,37 @@ # Changelog +## 1.14.1 - 10/24/20 +* bugfix where setting `_ok_code` to not include 0, but 0 was the exit code [#545](https://github.com/amoffat/sh/pull/545) + +## 1.14.0 - 8/28/20 +* `_env` now more lenient in accepting dictionary-like objects [#527](https://github.com/amoffat/sh/issues/527) +* `None` and `False` arguments now do not pass through to underlying command [#525](https://github.com/amoffat/sh/pull/525) +* Implemented `find_spec` on the fancy importer, which fixes some Python3.4+ issues [#536](https://github.com/amoffat/sh/pull/536) + +## 1.13.1 - 4/28/20 +* regression fix if `_fg=False` [#520](https://github.com/amoffat/sh/issues/520) + +## 1.13.0 - 4/27/20 +* minor Travis CI fixes [#492](https://github.com/amoffat/sh/pull/492) +* bugfix for boolean long options not respecting `_long_prefix` [#488](https://github.com/amoffat/sh/pull/488) +* fix deprecation warning on Python 3.6 regexes [#482](https://github.com/amoffat/sh/pull/482) +* `_pass_fds` and `_close_fds` special kwargs for controlling file descriptor inheritance in child. +* more efficiently closing inherited fds [#406](https://github.com/amoffat/sh/issues/406) +* bugfix where passing invalid dictionary to `_env` will cause a mysterious child 255 exit code. [#497](https://github.com/amoffat/sh/pull/497) +* bugfix where `_in` using 0 or `sys.stdin` wasn't behaving like a TTY, if it was in fact a TTY. [#514](https://github.com/amoffat/sh/issues/514) +* bugfix where `help(sh)` raised an exception [#455](https://github.com/amoffat/sh/issues/455) +* bugfix fixing broken interactive ssh tutorial from docs +* change to automatic tty merging into a single pty if `_tty_in=True` and `_tty_out=True` +* introducing `_unify_ttys`, default False, which allows explicit tty merging into single pty +* contrib command for `ssh` connections requiring passwords +* performance fix for polling output too fast when using `_iter` [#462](https://github.com/amoffat/sh/issues/462) +* execution contexts can now be used in python shell [#466](https://github.com/amoffat/sh/pull/466) +* bugfix `ErrorReturnCode` instances can now be pickled +* bugfix passing empty string or `None` for `_in` hanged [#427](https://github.com/amoffat/sh/pull/427) +* bugfix where passing a filename or file-like object to `_out` wasn't using os.dup2 [#449](https://github.com/amoffat/sh/issues/449) +* regression make `_fg` work with `_cwd` again [#330](https://github.com/amoffat/sh/issues/330) +* an invalid `_cwd` now raises a `ForkException` not an `OSError`. +* AIX support [#477](https://github.com/amoffat/sh/issues/477) +* added a `timeout=None` param to `RunningCommand.wait()` [#515](https://github.com/amoffat/sh/issues/515) ## 1.12.14 - 6/6/17 * bugfix for poor sleep performance [#378](https://github.com/amoffat/sh/issues/378) @@ -140,7 +173,7 @@ * Bugfix where input arguments were being assumed as ascii or unicode, but never as a string in a different encoding. * _long_sep keyword argument added joining together a dictionary of arguments passed in to a command * Commands can now be passed a dictionary of args, and the keys will be interpretted "raw", with no underscore-to-hyphen conversion -* Reserved Python keywords can now be used as subcommands by appending an underscore `_` to them +* Reserved Python keywords can now be used as subcommands by appending an underscore `_` to them ## 1.07 - 11/21/12 @@ -151,7 +184,7 @@ * Added `_decode_errors` to be passed to all stdout/stderr decoding of a process. * Added `_no_out`, `_no_err`, and `_no_pipe` special keyword arguments. These are used for long-running processes with lots of output. * Changed custom loggers that were created for each process to fixed loggers, so there are no longer logger references laying around in the logging module after the process ends and it garbage collected. - + ## 1.06 - 11/10/12 @@ -166,11 +199,11 @@ * Changing status from alpha to beta. * Python 3.3 officially supported. -* Documentation fix. The section on exceptions now references the fact that signals do not raise an exception, even for signals that might seem like they should, e.g. segfault. +* Documentation fix. The section on exceptions now references the fact that signals do not raise an exception, even for signals that might seem like they should, e.g. segfault. * Bugfix with Python 3.3 where importing commands from the sh namespace resulted in an error related to `__path__` * Long-form and short-form options to commands may now be given False to disable the option from being passed into the command. This is useful to pass in a boolean flag that you flip to either True or False to enable or disable some functionality at runtime. ## 1.04 - 10/07/12 -* Making `Command` class resolve the `path` parameter with `which` by default instead of expecting it to be resolved before it is passed in. This change shouldn't affect backwards compatibility. +* Making `Command` class resolve the `path` parameter with `which` by default instead of expecting it to be resolved before it is passed in. This change shouldn't affect backwards compatibility. * Fixing a bug when an exception is raised from a program, and the error output has non-ascii text. This didn't work in Python < 3.0, because .decode()'s default encoding is typically ascii. diff -Nru python-sh-1.12.14/.coveragerc python-sh-1.14.1/.coveragerc --- python-sh-1.12.14/.coveragerc 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/.coveragerc 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,10 @@ +[run] +branch = True +source = + sh.py + +[report] +exclude_lines = + pragma: no cover + if __name__ == .__main__.: + def __repr__ diff -Nru python-sh-1.12.14/debian/changelog python-sh-1.14.1/debian/changelog --- python-sh-1.12.14/debian/changelog 2019-08-09 09:34:27.000000000 +0000 +++ python-sh-1.14.1/debian/changelog 2020-12-11 16:58:09.000000000 +0000 @@ -1,3 +1,25 @@ +python-sh (1.14.1-1) unstable; urgency=medium + + [ Ondřej Nový ] + * Bump Standards-Version to 4.4.1. + * d/control: Update Maintainer field with new Debian Python Team + contact address. + * d/control: Update Vcs-* fields with new Debian Python Team Salsa + layout. + + [ Debian Janitor ] + * Set field Upstream-Name in debian/copyright. + * Set upstream metadata fields: Bug-Database, Bug-Submit, Repository, + Repository-Browse. + * Update standards version to 4.5.0, no changes needed. + + [ Gordon Ball ] + * d/watch: use github as upstream + * New upstream version 1.14.1 + * d/control: Rules-Requires-Root: no + + -- Gordon Ball Fri, 11 Dec 2020 16:58:09 +0000 + python-sh (1.12.14-2) unstable; urgency=medium * Team upload. diff -Nru python-sh-1.12.14/debian/control python-sh-1.14.1/debian/control --- python-sh-1.12.14/debian/control 2019-08-09 09:34:04.000000000 +0000 +++ python-sh-1.14.1/debian/control 2020-12-11 16:58:09.000000000 +0000 @@ -1,18 +1,19 @@ Source: python-sh Section: python Priority: optional -Maintainer: Debian Python Modules Team +Maintainer: Debian Python Team Uploaders: Tianon Gravi , Paul Tagliamonte Build-Depends: debhelper-compat (= 12), dh-python, python3-all (>= 3.1~), python3-setuptools -Standards-Version: 4.4.0 +Standards-Version: 4.5.0 Homepage: https://github.com/amoffat/sh/ -Vcs-Git: https://salsa.debian.org/python-team/modules/python-sh.git -Vcs-Browser: https://salsa.debian.org/python-team/modules/python-sh +Vcs-Git: https://salsa.debian.org/python-team/packages/python-sh.git +Vcs-Browser: https://salsa.debian.org/python-team/packages/python-sh Testsuite: autopkgtest-pkg-python +Rules-Requires-Root: no Package: python3-sh Architecture: all diff -Nru python-sh-1.12.14/debian/copyright python-sh-1.14.1/debian/copyright --- python-sh-1.12.14/debian/copyright 2019-08-09 09:17:32.000000000 +0000 +++ python-sh-1.14.1/debian/copyright 2020-12-11 16:58:09.000000000 +0000 @@ -1,5 +1,6 @@ Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Source: https://github.com/amoffat/sh +Upstream-Name: sh Files: * Copyright: 2011-2012 Andrew Moffat diff -Nru python-sh-1.12.14/debian/upstream/metadata python-sh-1.14.1/debian/upstream/metadata --- python-sh-1.12.14/debian/upstream/metadata 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/debian/upstream/metadata 2020-12-11 16:58:09.000000000 +0000 @@ -0,0 +1,4 @@ +Bug-Database: https://github.com/amoffat/sh/issues +Bug-Submit: https://github.com/amoffat/sh/issues/new +Repository: https://github.com/amoffat/sh.git +Repository-Browse: https://github.com/amoffat/sh diff -Nru python-sh-1.12.14/debian/watch python-sh-1.14.1/debian/watch --- python-sh-1.12.14/debian/watch 2019-08-09 09:17:32.000000000 +0000 +++ python-sh-1.14.1/debian/watch 2020-12-11 16:58:09.000000000 +0000 @@ -1,3 +1,3 @@ -version=3 +version=4 opts=uversionmangle=s/(rc|a|b|c)/~$1/ \ -https://pypi.debian.net/sh/sh-(.+)\.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz))) +https://github.com/amoffat/sh/tags .*/archive/@ANY_VERSION@@ARCHIVE_EXT@ diff -Nru python-sh-1.12.14/docker_test_suite/build.sh python-sh-1.14.1/docker_test_suite/build.sh --- python-sh-1.12.14/docker_test_suite/build.sh 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/docker_test_suite/build.sh 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,4 @@ +#!/bin/bash +set -ex +cp ../requirements-dev.txt . +docker build -t amoffat/shtest $@ . diff -Nru python-sh-1.12.14/docker_test_suite/Dockerfile python-sh-1.14.1/docker_test_suite/Dockerfile --- python-sh-1.12.14/docker_test_suite/Dockerfile 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/docker_test_suite/Dockerfile 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,47 @@ +FROM ubuntu:bionic + +ARG cache_bust +RUN apt-get update +RUN apt-get -y install locales + +RUN locale-gen en_US.UTF-8 +ENV LANG en_US.UTF-8 +ENV LANGUAGE en_US:en +ENV LC_ALL en_US.UTF-8 + +RUN apt-get -y install\ + software-properties-common\ + curl\ + sudo\ + python\ + lsof + +RUN add-apt-repository ppa:deadsnakes/ppa +RUN apt-get update +RUN apt-get -y install\ + python2.6\ + python2.7\ + python3.1\ + python3.2\ + python3.3\ + python3.4\ + python3.5\ + python3.6\ + python3.7\ + python3.8 + +RUN apt-get -y install python3-distutils\ + && curl https://bootstrap.pypa.io/get-pip.py | python - + +ARG uid=1000 +RUN groupadd -g $uid shtest\ + && useradd -m -u $uid -g $uid shtest\ + && gpasswd -a shtest sudo\ + && echo "shtest:shtest" | chpasswd + +COPY requirements-dev.txt /tmp/ +RUN pip install -r /tmp/requirements-dev.txt + +USER shtest +WORKDIR /home/shtest/sh +ENTRYPOINT ["python", "sh.py", "test"] diff -Nru python-sh-1.12.14/docker_test_suite/run.sh python-sh-1.14.1/docker_test_suite/run.sh --- python-sh-1.12.14/docker_test_suite/run.sh 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/docker_test_suite/run.sh 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,3 @@ +#!/bin/bash +set -ex +docker run -it --rm -v $(pwd)/../:/home/shtest/sh amoffat/shtest $@ diff -Nru python-sh-1.12.14/docker_test_suite/shell.sh python-sh-1.14.1/docker_test_suite/shell.sh --- python-sh-1.12.14/docker_test_suite/shell.sh 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/docker_test_suite/shell.sh 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,3 @@ +#!/bin/bash +set -ex +docker run -it --rm -v $(pwd)/../:/home/shtest/sh --entrypoint=/bin/bash amoffat/shtest $@ Binary files /tmp/tmpTTqAMb/HtD7nfrKOo/python-sh-1.12.14/gitads.png and /tmp/tmpTTqAMb/HQYMJq2Ijw/python-sh-1.14.1/gitads.png differ diff -Nru python-sh-1.12.14/.github/FUNDING.yml python-sh-1.14.1/.github/FUNDING.yml --- python-sh-1.12.14/.github/FUNDING.yml 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/.github/FUNDING.yml 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,12 @@ +# These are supported funding model platforms + +github: [amoffat] +patreon: # Replace with a single Patreon username +open_collective: # Replace with a single Open Collective username +ko_fi: # Replace with a single Ko-fi username +tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel +community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry +liberapay: # Replace with a single Liberapay username +issuehunt: # Replace with a single IssueHunt username +otechie: # Replace with a single Otechie username +custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2'] diff -Nru python-sh-1.12.14/.gitignore python-sh-1.14.1/.gitignore --- python-sh-1.12.14/.gitignore 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/.gitignore 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,4 @@ +__pycache__/ +*.py[co] +.tox +.coverage Binary files /tmp/tmpTTqAMb/HtD7nfrKOo/python-sh-1.12.14/logo-230.png and /tmp/tmpTTqAMb/HQYMJq2Ijw/python-sh-1.14.1/logo-230.png differ Binary files /tmp/tmpTTqAMb/HtD7nfrKOo/python-sh-1.12.14/logo-big.png and /tmp/tmpTTqAMb/HQYMJq2Ijw/python-sh-1.14.1/logo-big.png differ diff -Nru python-sh-1.12.14/PKG-INFO python-sh-1.14.1/PKG-INFO --- python-sh-1.12.14/PKG-INFO 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/PKG-INFO 1970-01-01 00:00:00.000000000 +0000 @@ -1,114 +0,0 @@ -Metadata-Version: 1.1 -Name: sh -Version: 1.12.14 -Summary: Python subprocess replacement -Home-page: https://github.com/amoffat/sh -Author: Andrew Moffat -Author-email: andrew.robert.moffat@gmail.com -License: MIT -Description: .. image:: https://raw.githubusercontent.com/amoffat/sh/master/logo-230.png - :target: https://amoffat.github.com/sh - :alt: Logo - - | - - .. image:: https://img.shields.io/pypi/v/sh.svg?style=flat-square - :target: https://pypi.python.org/pypi/sh - :alt: Version - .. image:: https://img.shields.io/pypi/pyversions/sh.svg?style=flat-square - :target: https://pypi.python.org/pypi/sh - :alt: Python Versions - .. image:: https://img.shields.io/travis/amoffat/sh/master.svg?style=flat-square - :target: https://travis-ci.org/amoffat/sh - :alt: Build Status - .. image:: https://img.shields.io/coveralls/amoffat/sh.svg?style=flat-square - :target: https://coveralls.io/r/amoffat/sh?branch=master - :alt: Coverage Status - - | - - sh is a full-fledged subprocess replacement for Python 2.6 - 3.6, PyPy and PyPy3 - that allows you to call any program as if it were a function: - - .. code:: python - - from sh import ifconfig - print ifconfig("eth0") - - sh is *not* a collection of system commands implemented in Python. - - `Complete documentation here`_ - - Installation - ============ - - :: - - $> pip install sh - - Updating the docs - ================= - - Check out the `gh-pages `_ branch and follow the ``README.rst`` there. - - Developers - ========== - - Testing - ------- - - First install the development requirements:: - - $> pip install -r requirements-dev.txt - - The run the tests for all Python versions on your system:: - - $> python sh.py test - - To run a single test for all environments:: - - $> python sh.py test FunctionalTests.test_unicode_arg - - To run a single test for a single environment:: - - $> python sh.py test -e 3.4 FunctionalTests.test_unicode_arg - - Coverage - -------- - - First run all of the tests:: - - $> python sh.py test - - This will aggregate a ``.coverage``. You may then visualize the report with:: - - $> coverage report - - Or generate visual html files with:: - - $> coverage html - - Which will create ``./htmlcov/index.html`` that you may open in a web browser. - -Keywords: subprocess,process,shell,launch,program -Platform: UNKNOWN -Classifier: Development Status :: 5 - Production/Stable -Classifier: Environment :: Console -Classifier: Intended Audience :: Developers -Classifier: Intended Audience :: System Administrators -Classifier: License :: OSI Approved :: MIT License -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 2 -Classifier: Programming Language :: Python :: 2.6 -Classifier: Programming Language :: Python :: 2.7 -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.1 -Classifier: Programming Language :: Python :: 3.2 -Classifier: Programming Language :: Python :: 3.3 -Classifier: Programming Language :: Python :: 3.4 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Classifier: Topic :: Software Development :: Build Tools -Classifier: Topic :: Software Development :: Libraries :: Python Modules diff -Nru python-sh-1.12.14/README.rst python-sh-1.14.1/README.rst --- python-sh-1.12.14/README.rst 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/README.rst 2020-10-24 17:34:57.000000000 +0000 @@ -7,6 +7,9 @@ .. image:: https://img.shields.io/pypi/v/sh.svg?style=flat-square :target: https://pypi.python.org/pypi/sh :alt: Version +.. image:: https://img.shields.io/pypi/dm/sh.svg?style=flat-square + :target: https://pypi.python.org/pypi/sh + :alt: Downloads Status .. image:: https://img.shields.io/pypi/pyversions/sh.svg?style=flat-square :target: https://pypi.python.org/pypi/sh :alt: Python Versions @@ -19,17 +22,17 @@ | -sh is a full-fledged subprocess replacement for Python 2.6 - 3.6, PyPy and PyPy3 -that allows you to call any program as if it were a function: +sh is a full-fledged subprocess replacement for Python 2.6 - 3.8, PyPy and PyPy3 +that allows you to call *any* program as if it were a function: .. code:: python from sh import ifconfig - print ifconfig("eth0") + print(ifconfig("eth0")) sh is *not* a collection of system commands implemented in Python. -`Complete documentation here`_ +`Complete documentation here `_ Installation ============ @@ -37,33 +40,42 @@ :: $> pip install sh + +Support +======= +* `Andrew Moffat `_ - author/maintainer +* `Erik Cederstrand `_ - maintainer -Updating the docs -================= - -Check out the `gh-pages `_ branch and follow the ``README.rst`` there. Developers ========== +Updating the docs +----------------- + +Check out the `gh-pages `_ branch and follow the ``README.rst`` there. + Testing ------- -First install the development requirements:: +I've included a Docker test suite in the `docker_test_suit/` folder. To build the image, `cd` into that directory and +run:: - $> pip install -r requirements-dev.txt + $> ./build.sh -The run the tests for all Python versions on your system:: +This will install ubuntu 18.04 LTS and all python versions from 2.6-3.8. Once it's done, stay in that directory and +run:: - $> python sh.py test + $> ./run.sh -To run a single test for all environments:: +This will mount your local code directory into the container and start the test suite, which will take a long time to +run. If you wish to run a single test, you may pass that test to `./run.sh`:: - $> python sh.py test FunctionalTests.test_unicode_arg + $> ./run.sh FunctionalTests.test_unicode_arg To run a single test for a single environment:: - $> python sh.py test -e 3.4 FunctionalTests.test_unicode_arg + $> ./run.sh -e 3.4 FunctionalTests.test_unicode_arg Coverage -------- diff -Nru python-sh-1.12.14/requirements-dev.txt python-sh-1.14.1/requirements-dev.txt --- python-sh-1.12.14/requirements-dev.txt 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/requirements-dev.txt 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,6 @@ +Pygments==2.1.3 +coverage==4.2 +coveralls==1.1 +docopt==0.6.2 +docutils==0.12 +flake8==3.7.9 diff -Nru python-sh-1.12.14/requirements-docs.txt python-sh-1.14.1/requirements-docs.txt --- python-sh-1.12.14/requirements-docs.txt 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/requirements-docs.txt 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,12 @@ +alabaster==0.7.9 +Babel==2.3.4 +docutils==0.12 +imagesize==0.7.1 +Jinja2==2.10.3 +MarkupSafe==0.23 +Pygments==2.1.3 +pytz==2016.7 +six==1.10.0 +snowballstemmer==1.2.1 +Sphinx==3.1.1 +sphinx-rtd-theme==0.5.0 diff -Nru python-sh-1.12.14/setup.cfg python-sh-1.14.1/setup.cfg --- python-sh-1.12.14/setup.cfg 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/setup.cfg 2020-10-24 17:34:57.000000000 +0000 @@ -4,8 +4,5 @@ [metadata] license_file = LICENSE.txt -[egg_info] -tag_build = -tag_date = 0 -tag_svn_revision = 0 - +[flake8] +max-line-length = 120 diff -Nru python-sh-1.12.14/setup.py python-sh-1.14.1/setup.py --- python-sh-1.12.14/setup.py 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/setup.py 2020-10-24 17:34:57.000000000 +0000 @@ -10,7 +10,7 @@ HERE = dirname(abspath(__file__)) author = "Andrew Moffat" -author_email = "andrew.robert.moffat@gmail.com" +author_email = "arwmoffat@gmail.com" keywords = ["subprocess", "process", "shell", "launch", "program"] @@ -48,6 +48,8 @@ "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", + "Programming Language :: Python :: 3.7", + "Programming Language :: Python :: 3.8", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", "Topic :: Software Development :: Build Tools", diff -Nru python-sh-1.12.14/sh.egg-info/dependency_links.txt python-sh-1.14.1/sh.egg-info/dependency_links.txt --- python-sh-1.12.14/sh.egg-info/dependency_links.txt 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/sh.egg-info/dependency_links.txt 1970-01-01 00:00:00.000000000 +0000 @@ -1 +0,0 @@ - diff -Nru python-sh-1.12.14/sh.egg-info/pbr.json python-sh-1.14.1/sh.egg-info/pbr.json --- python-sh-1.12.14/sh.egg-info/pbr.json 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/sh.egg-info/pbr.json 1970-01-01 00:00:00.000000000 +0000 @@ -1 +0,0 @@ -{"is_release": false, "git_version": "93eca5d"} \ No newline at end of file diff -Nru python-sh-1.12.14/sh.egg-info/PKG-INFO python-sh-1.14.1/sh.egg-info/PKG-INFO --- python-sh-1.12.14/sh.egg-info/PKG-INFO 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/sh.egg-info/PKG-INFO 1970-01-01 00:00:00.000000000 +0000 @@ -1,114 +0,0 @@ -Metadata-Version: 1.1 -Name: sh -Version: 1.12.14 -Summary: Python subprocess replacement -Home-page: https://github.com/amoffat/sh -Author: Andrew Moffat -Author-email: andrew.robert.moffat@gmail.com -License: MIT -Description: .. image:: https://raw.githubusercontent.com/amoffat/sh/master/logo-230.png - :target: https://amoffat.github.com/sh - :alt: Logo - - | - - .. image:: https://img.shields.io/pypi/v/sh.svg?style=flat-square - :target: https://pypi.python.org/pypi/sh - :alt: Version - .. image:: https://img.shields.io/pypi/pyversions/sh.svg?style=flat-square - :target: https://pypi.python.org/pypi/sh - :alt: Python Versions - .. image:: https://img.shields.io/travis/amoffat/sh/master.svg?style=flat-square - :target: https://travis-ci.org/amoffat/sh - :alt: Build Status - .. image:: https://img.shields.io/coveralls/amoffat/sh.svg?style=flat-square - :target: https://coveralls.io/r/amoffat/sh?branch=master - :alt: Coverage Status - - | - - sh is a full-fledged subprocess replacement for Python 2.6 - 3.6, PyPy and PyPy3 - that allows you to call any program as if it were a function: - - .. code:: python - - from sh import ifconfig - print ifconfig("eth0") - - sh is *not* a collection of system commands implemented in Python. - - `Complete documentation here`_ - - Installation - ============ - - :: - - $> pip install sh - - Updating the docs - ================= - - Check out the `gh-pages `_ branch and follow the ``README.rst`` there. - - Developers - ========== - - Testing - ------- - - First install the development requirements:: - - $> pip install -r requirements-dev.txt - - The run the tests for all Python versions on your system:: - - $> python sh.py test - - To run a single test for all environments:: - - $> python sh.py test FunctionalTests.test_unicode_arg - - To run a single test for a single environment:: - - $> python sh.py test -e 3.4 FunctionalTests.test_unicode_arg - - Coverage - -------- - - First run all of the tests:: - - $> python sh.py test - - This will aggregate a ``.coverage``. You may then visualize the report with:: - - $> coverage report - - Or generate visual html files with:: - - $> coverage html - - Which will create ``./htmlcov/index.html`` that you may open in a web browser. - -Keywords: subprocess,process,shell,launch,program -Platform: UNKNOWN -Classifier: Development Status :: 5 - Production/Stable -Classifier: Environment :: Console -Classifier: Intended Audience :: Developers -Classifier: Intended Audience :: System Administrators -Classifier: License :: OSI Approved :: MIT License -Classifier: Programming Language :: Python -Classifier: Programming Language :: Python :: 2 -Classifier: Programming Language :: Python :: 2.6 -Classifier: Programming Language :: Python :: 2.7 -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.1 -Classifier: Programming Language :: Python :: 3.2 -Classifier: Programming Language :: Python :: 3.3 -Classifier: Programming Language :: Python :: 3.4 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: Implementation :: CPython -Classifier: Programming Language :: Python :: Implementation :: PyPy -Classifier: Topic :: Software Development :: Build Tools -Classifier: Topic :: Software Development :: Libraries :: Python Modules diff -Nru python-sh-1.12.14/sh.egg-info/SOURCES.txt python-sh-1.14.1/sh.egg-info/SOURCES.txt --- python-sh-1.12.14/sh.egg-info/SOURCES.txt 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/sh.egg-info/SOURCES.txt 1970-01-01 00:00:00.000000000 +0000 @@ -1,13 +0,0 @@ -CHANGELOG.md -LICENSE.txt -MANIFEST.in -README.rst -setup.cfg -setup.py -sh.py -test.py -sh.egg-info/PKG-INFO -sh.egg-info/SOURCES.txt -sh.egg-info/dependency_links.txt -sh.egg-info/pbr.json -sh.egg-info/top_level.txt \ No newline at end of file diff -Nru python-sh-1.12.14/sh.egg-info/top_level.txt python-sh-1.14.1/sh.egg-info/top_level.txt --- python-sh-1.12.14/sh.egg-info/top_level.txt 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/sh.egg-info/top_level.txt 1970-01-01 00:00:00.000000000 +0000 @@ -1 +0,0 @@ -sh diff -Nru python-sh-1.12.14/sh.py python-sh-1.14.1/sh.py --- python-sh-1.12.14/sh.py 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/sh.py 2020-10-24 17:34:57.000000000 +0000 @@ -1,8 +1,8 @@ """ http://amoffat.github.io/sh/ """ -#=============================================================================== -# Copyright (C) 2011-2017 by Andrew Moffat +# =============================================================================== +# Copyright (C) 2011-2020 by Andrew Moffat # # Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the "Software"), to deal @@ -21,58 +21,55 @@ # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN # THE SOFTWARE. -#=============================================================================== - - -__version__ = "1.12.14" +# =============================================================================== +__version__ = "1.14.1" __project_url__ = "https://github.com/amoffat/sh" - +from collections import deque +try: + from collections.abc import Mapping +except ImportError: + from collections import Mapping +from contextlib import contextmanager +from functools import partial +from io import UnsupportedOperation, open as fdopen +from locale import getpreferredencoding +from types import ModuleType, GeneratorType +import ast +import errno +import fcntl +import gc +import getpass +import glob as glob_module +import inspect +import logging +import os import platform - -if "windows" in platform.system().lower(): # pragma: no cover - raise ImportError("sh %s is currently only supported on linux and osx. \ -please install pbs 0.110 (http://pypi.python.org/pypi/pbs) for windows \ -support." % __version__) - - +import pty +import pwd +import re +import select +import signal +import stat +import struct import sys +import termios +import threading +import time +import traceback +import tty +import warnings +import weakref + IS_PY3 = sys.version_info[0] == 3 MINOR_VER = sys.version_info[1] IS_PY26 = sys.version_info[0] == 2 and MINOR_VER == 6 - -import traceback -import os -import re -import time -import getpass -from types import ModuleType, GeneratorType -from functools import partial -import inspect -import tempfile -import stat -import glob as glob_module -import ast -from contextlib import contextmanager -import pwd -import errno -from io import UnsupportedOperation, open as fdopen - -from locale import getpreferredencoding -DEFAULT_ENCODING = getpreferredencoding() or "UTF-8" - -# normally i would hate this idea of using a global to signify whether we are -# running tests, because it breaks the assumption that what is running in the -# tests is what will run live, but we ONLY use this in a place that has no -# serious side-effects that could change anything. as long as we do that, it -# should be ok -RUNNING_TESTS = bool(int(os.environ.get("SH_TESTS_RUNNING", "0"))) -FORCE_USE_SELECT = bool(int(os.environ.get("SH_TESTS_USE_SELECT", "0"))) - if IS_PY3: from io import StringIO + ioStringIO = StringIO from io import BytesIO as cStringIO + iocStringIO = cStringIO from queue import Queue, Empty @@ -87,33 +84,35 @@ from io import BytesIO as iocStringIO from Queue import Queue, Empty -IS_OSX = platform.system() == "Darwin" -THIS_DIR = os.path.dirname(os.path.realpath(__file__)) -SH_LOGGER_NAME = __name__ +try: + from shlex import quote as shlex_quote # here from 3.3 onward +except ImportError: + from pipes import quote as shlex_quote # undocumented before 2.7 +if "windows" in platform.system().lower(): # pragma: no cover + raise ImportError("sh %s is currently only supported on linux and osx. \ +please install pbs 0.110 (http://pypi.python.org/pypi/pbs) for windows \ +support." % __version__) -import errno -import pty -import termios -import signal -import gc -import select -import threading -import tty -import fcntl -import struct -import resource -from collections import deque -import logging -import weakref +DEFAULT_ENCODING = getpreferredencoding() or "UTF-8" +IS_MACOS = platform.system() in ("AIX", "Darwin") +THIS_DIR = os.path.dirname(os.path.realpath(__file__)) +SH_LOGGER_NAME = __name__ + +# normally i would hate this idea of using a global to signify whether we are +# running tests, because it breaks the assumption that what is running in the +# tests is what will run live, but we ONLY use this in a place that has no +# serious side-effects that could change anything. as long as we do that, it +# should be ok +RUNNING_TESTS = bool(int(os.environ.get("SH_TESTS_RUNNING", "0"))) +FORCE_USE_SELECT = bool(int(os.environ.get("SH_TESTS_USE_SELECT", "0"))) # a re-entrant lock for pushd. this way, multiple threads that happen to use # pushd will all see the current working directory for the duration of the # with-context PUSHD_LOCK = threading.RLock() - if hasattr(inspect, "getfullargspec"): def get_num_args(fn): return len(inspect.getfullargspec(fn).args) @@ -127,17 +126,14 @@ basestring = str long = int - _unicode_methods = set(dir(unicode())) - HAS_POLL = hasattr(select, "poll") POLLER_EVENT_READ = 1 POLLER_EVENT_WRITE = 2 POLLER_EVENT_HUP = 4 POLLER_EVENT_ERROR = 8 - # here we use an use a poller interface that transparently selects the most # capable poller (out of either select.select or select.poll). this was added # by zhangyafeikimi when he discovered that if the fds created internally by sh @@ -211,9 +207,9 @@ f = self._get_file_object(fd) if events & (select.POLLIN | select.POLLPRI): results.append((f, POLLER_EVENT_READ)) - elif events & (select.POLLOUT): + elif events & select.POLLOUT: results.append((f, POLLER_EVENT_WRITE)) - elif events & (select.POLLHUP): + elif events & select.POLLHUP: results.append((f, POLLER_EVENT_HUP)) elif events & (select.POLLERR | select.POLLNVAL): results.append((f, POLLER_EVENT_ERROR)) @@ -231,13 +227,15 @@ def __len__(self): return len(self.rlist) + len(self.wlist) + len(self.xlist) - def _register(self, f, l): - if f not in l: - l.append(f) - - def _unregister(self, f, l): - if f in l: - l.remove(f) + @staticmethod + def _register(f, events): + if f not in events: + events.append(f) + + @staticmethod + def _unregister(f, events): + if f in events: + events.remove(f) def register_read(self, f): self._register(f, self.rlist) @@ -296,7 +294,7 @@ # but attempt to fall back to something try: s = s.encode(DEFAULT_ENCODING) - except: + except UnicodeEncodeError: s = s.encode(fallback_encoding, "replace") return s @@ -330,6 +328,7 @@ program we're testing throws may not be the same class that we pass to assertRaises """ + def __subclasscheck__(self, o): other_bases = set([b.__name__ for b in o.__bases__]) return self.__name__ in other_bases or o.__name__ == self.__name__ @@ -343,27 +342,31 @@ derived classes with the format: ErrorReturnCode_NNN where NNN is the exit code number. the reason for this is it reduces boiler plate code when testing error return codes: - + try: some_cmd() except ErrorReturnCode_12: print("couldn't do X") - + vs: try: some_cmd() except ErrorReturnCode as e: if e.exit_code == 12: print("couldn't do X") - + it's not much of a savings, but i believe it makes the code easier to read """ truncate_cap = 750 + def __reduce__(self): + return self.__class__, (self.full_cmd, self.stdout, self.stderr, self.truncate) + def __init__(self, full_cmd, stdout, stderr, truncate=True): self.full_cmd = full_cmd self.stdout = stdout self.stderr = stderr + self.truncate = truncate exc_stdout = self.stdout if truncate: @@ -387,17 +390,29 @@ stderr=exc_stderr.decode(DEFAULT_ENCODING, "replace") ) + if not IS_PY3: + # Exception messages should be treated as an API which takes native str type on both + # Python2 and Python3. (Meaning, it's a byte string on Python2 and a text string on + # Python3) + msg = encode_to_py3bytes_or_py2str(msg) + super(ErrorReturnCode, self).__init__(msg) -class SignalException(ErrorReturnCode): pass +class SignalException(ErrorReturnCode): + pass + + class TimeoutException(Exception): """ the exception thrown when a command is killed because a specified - timeout (via _timeout) was hit """ - def __init__(self, exit_code): + timeout (via _timeout or .wait(timeout)) was hit """ + + def __init__(self, exit_code, full_cmd): self.exit_code = exit_code + self.full_cmd = full_cmd super(Exception, self).__init__() + SIGNALS_THAT_SHOULD_THROW_EXCEPTION = set(( signal.SIGABRT, signal.SIGBUS, @@ -416,18 +431,14 @@ # we subclass AttributeError because: # https://github.com/ipython/ipython/issues/2577 # https://github.com/amoffat/sh/issues/97#issuecomment-10610629 -class CommandNotFound(AttributeError): pass - - +class CommandNotFound(AttributeError): + pass -rc_exc_regex = re.compile("(ErrorReturnCode|SignalException)_((\d+)|SIG[a-zA-Z]+)") +rc_exc_regex = re.compile(r"(ErrorReturnCode|SignalException)_((\d+)|SIG[a-zA-Z]+)") rc_exc_cache = {} -SIGNAL_MAPPING = {} -for k,v in signal.__dict__.items(): - if re.match(r"SIG[a-zA-Z]+", k): - SIGNAL_MAPPING[v] = k +SIGNAL_MAPPING = dict([(v, k) for k, v in signal.__dict__.items() if re.match(r"SIG[a-zA-Z]+", k)]) def get_exc_from_name(name): @@ -475,7 +486,7 @@ except KeyError: pass - if rc > 0: + if rc >= 0: name = "ErrorReturnCode_%d" % rc base = ErrorReturnCode else: @@ -488,7 +499,6 @@ return exc - # we monkey patch glob. i'm normally generally against monkey patching, but i # decided to do this really un-intrusive patch because we need a way to detect # if a list that we pass into an sh command was generated from glob. the reason @@ -505,18 +515,23 @@ # wiser, but we'll have results that we can make some determinations on _old_glob = glob_module.glob + class GlobResults(list): def __init__(self, path, results): self.path = path list.__init__(self, results) + def glob(path, *args, **kwargs): expanded = GlobResults(path, _old_glob(path, *args, **kwargs)) return expanded + glob_module.glob = glob +def canonicalize(path): + return os.path.abspath(os.path.expanduser(path)) def which(program, paths=None): @@ -525,10 +540,10 @@ specified, it is the entire list of search paths, and the PATH env is not used at all. otherwise, PATH env is used to look for the program """ - def is_exe(fpath): - return (os.path.exists(fpath) and - os.access(fpath, os.X_OK) and - os.path.isfile(os.path.realpath(fpath))) + def is_exe(file_path): + return (os.path.exists(file_path) and + os.access(file_path, os.X_OK) and + os.path.isfile(os.path.realpath(file_path))) found_path = None fpath, fname = os.path.split(program) @@ -536,7 +551,7 @@ # if there's a path component, then we've specified a path to the program, # and we should just test if that program is executable. if it is, return if fpath: - program = os.path.abspath(os.path.expanduser(program)) + program = canonicalize(program) if is_exe(program): found_path = program @@ -552,7 +567,7 @@ paths_to_search.extend(env_paths) for path in paths_to_search: - exe_file = os.path.join(path, program) + exe_file = os.path.join(canonicalize(path), program) if is_exe(exe_file): found_path = exe_file break @@ -584,8 +599,6 @@ return cmd - - class Logger(object): """ provides a memory-inexpensive logger. a gotcha about python's builtin logger is that logger objects are never garbage collected. if you create a @@ -593,7 +606,7 @@ script is done. with sh, it's easy to create loggers with unique names if we want our loggers to include our command arguments. for example, these are all unique loggers: - + ls -l ls -l /tmp ls /tmp @@ -603,38 +616,39 @@ "context", which will be the very unique name. this allows us to get a logger with a very general name, eg: "command", and have a unique name appended to it via the context, eg: "ls -l /tmp" """ + def __init__(self, name, context=None): self.name = name self.log = logging.getLogger("%s.%s" % (SH_LOGGER_NAME, name)) - self.set_context(context) + self.context = self.sanitize_context(context) - def _format_msg(self, msg, *args): + def _format_msg(self, msg, *a): if self.context: msg = "%s: %s" % (self.context, msg) - return msg % args + return msg % a - def set_context(self, context): + @staticmethod + def sanitize_context(context): if context: context = context.replace("%", "%%") - self.context = context or "" + return context or "" def get_child(self, name, context): new_name = self.name + "." + name new_context = self.context + "." + context - l = Logger(new_name, new_context) - return l + return Logger(new_name, new_context) - def info(self, msg, *args): - self.log.info(self._format_msg(msg, *args)) + def info(self, msg, *a): + self.log.info(self._format_msg(msg, *a)) - def debug(self, msg, *args): - self.log.debug(self._format_msg(msg, *args)) + def debug(self, msg, *a): + self.log.debug(self._format_msg(msg, *a)) - def error(self, msg, *args): - self.log.error(self._format_msg(msg, *args)) + def error(self, msg, *a): + self.log.error(self._format_msg(msg, *a)) - def exception(self, msg, *args): - self.log.exception(self._format_msg(msg, *args)) + def exception(self, msg, *a): + self.log.exception(self._format_msg(msg, *a)) def default_logger_str(cmd, call_args, pid=None): @@ -645,7 +659,6 @@ return s - class RunningCommand(object): """ this represents an executing Command object. it is returned as the result of __call__() being executed on a Command instance. this creates a @@ -660,7 +673,7 @@ finishes, RunningCommand is smart enough to translate exit codes to exceptions. """ - # these are attributes that we allow to passthrough to OProc for + # these are attributes that we allow to pass through to OProc _OProc_attr_whitelist = set(( "signal", "terminate", @@ -679,8 +692,7 @@ def __init__(self, cmd, call_args, stdin, stdout, stderr): """ - cmd is an array, where each element is encoded as bytes (PY3) or str - (PY2) + cmd is a list, where each element is encoded as bytes (PY3) or str (PY2) """ # self.ran is used for auditing what actually ran. for example, in @@ -692,13 +704,13 @@ # arguments are the encoding we pass into _encoding, which falls back to # the system's encoding enc = call_args["encoding"] - self.ran = " ".join([arg.decode(enc, "ignore") for arg in cmd]) + self.ran = " ".join([shlex_quote(arg.decode(enc, "ignore")) for arg in cmd]) self.call_args = call_args self.cmd = cmd self.process = None - self._process_completed = False + self._waited_until_completion = False should_wait = True spawn_process = True @@ -713,7 +725,6 @@ spawn_process = False get_prepend_stack().append(self) - if call_args["piped"] or call_args["iter"] or call_args["iter_noblock"]: should_wait = False @@ -728,8 +739,7 @@ done_callback = call_args["done"] if done_callback: - call_args["done"] = partial(done_callback, self) - + call_args["done"] = partial(done_callback, self) # set up which stream should write to the pipe # TODO, make pipe None by default and limit the size of the Queue @@ -753,7 +763,7 @@ logger_str = log_str_factory(self.ran, call_args) self.log = Logger("command", logger_str) - self.log.info("starting process") + self.log.debug("starting process") if should_wait: self._spawned_and_waited = True @@ -764,33 +774,72 @@ process_assign_lock = threading.Lock() with process_assign_lock: self.process = OProc(self, self.log, cmd, stdin, stdout, stderr, - self.call_args, pipe, process_assign_lock) + self.call_args, pipe, process_assign_lock) logger_str = log_str_factory(self.ran, call_args, self.process.pid) - self.log.set_context(logger_str) + self.log.context = self.log.sanitize_context(logger_str) self.log.info("process started") if should_wait: self.wait() - - def wait(self): + def wait(self, timeout=None): """ waits for the running command to finish. this is called on all running commands, eventually, except for ones that run in the background + + if timeout is a number, it is the number of seconds to wait for the process to resolve. otherwise block on wait. + + this function can raise a TimeoutException, either because of a `_timeout` on the command itself as it was + launched, or because of a timeout passed into this method. """ - if not self._process_completed: - self._process_completed = True + if not self._waited_until_completion: + + # if we've been given a timeout, we need to poll is_alive() + if timeout is not None: + waited_for = 0 + sleep_amt = 0.1 + alive = False + exit_code = None + if timeout < 0: + raise RuntimeError("timeout cannot be negative") + + # while we still have time to wait, run this loop + # notice that alive and exit_code are only defined in this loop, but the loop is also guaranteed to run, + # defining them, given the constraints that timeout is non-negative + while waited_for <= timeout: + alive, exit_code = self.process.is_alive() + + # if we're alive, we need to wait some more, but let's sleep before we poll again + if alive: + time.sleep(sleep_amt) + waited_for += sleep_amt + + # but if we're not alive, we're done waiting + else: + break + + # if we've made it this far, and we're still alive, then it means we timed out waiting + if alive: + raise TimeoutException(None, self.ran) + + # if we didn't time out, we fall through and let the rest of the code handle exit_code. + # notice that we set _waited_until_completion here, only if we didn't time out. this allows us to + # re-wait again on timeout, if we catch the TimeoutException in the parent frame + self._waited_until_completion = True + + else: + exit_code = self.process.wait() + self._waited_until_completion = True - exit_code = self.process.wait() if self.process.timed_out: # if we timed out, our exit code represents a signal, which is # negative, so let's make it positive to store in our # TimeoutException - raise TimeoutException(-exit_code) + raise TimeoutException(-exit_code, self.ran) else: self.handle_command_exit_code(exit_code) - + # if an iterable command is using an instance of OProc for its stdin, # wait on it. the process is probably set to "piped", which means it # won't be waited on, which means exceptions won't propagate up to the @@ -798,23 +847,23 @@ if self.process._stdin_process: self.process._stdin_process.command.wait() - self.log.info("process completed") + self.log.debug("process completed") return self + def is_alive(self): + """ returns whether or not we're still alive. this call has side-effects on OProc """ + return self.process.is_alive()[0] def handle_command_exit_code(self, code): """ here we determine if we had an exception, or an error code that we weren't expecting to see. if we did, we create and raise an exception """ ca = self.call_args - exc_class = get_exc_exit_code_would_raise(code, ca["ok_code"], - ca["piped"]) + exc_class = get_exc_exit_code_would_raise(code, ca["ok_code"], ca["piped"]) if exc_class: - exc = exc_class(self.ran, self.process.stdout, self.process.stderr, - ca["truncate_exc"]) + exc = exc_class(self.ran, self.process.stdout, self.process.stderr, ca["truncate_exc"]) raise exc - @property def stdout(self): self.wait() @@ -830,14 +879,13 @@ self.wait() return self.process.exit_code - def __len__(self): return len(str(self)) def __enter__(self): """ we don't actually do anything here because anything that should have been done would have been done in the Command.__call__ call. - essentially all that has to happen is the comand be pushed on the + essentially all that has to happen is the command be pushed on the prepend stack. """ pass @@ -854,7 +902,7 @@ # so the slight timeout allows for that. while True: try: - chunk = self.process._pipe_queue.get(True, 0.001) + chunk = self.process._pipe_queue.get(True, self.call_args["iter_poll_time"]) except Empty: if self.call_args["iter_noblock"]: return errno.EWOULDBLOCK @@ -864,16 +912,14 @@ self._stopped_iteration = True raise StopIteration() try: - return chunk.decode(self.call_args["encoding"], - self.call_args["decode_errors"]) + return chunk.decode(self.call_args["encoding"], self.call_args["decode_errors"]) except UnicodeDecodeError: return chunk - # python 3 __next__ = next - def __exit__(self, typ, value, traceback): + def __exit__(self, exc_type, exc_val, exc_tb): if self.call_args["with"] and get_prepend_stack(): get_prepend_stack().pop() @@ -889,8 +935,7 @@ """ a magic method defined for python2. calling unicode() on a RunningCommand object will call this """ if self.process and self.stdout: - return self.stdout.decode(self.call_args["encoding"], - self.call_args["decode_errors"]) + return self.stdout.decode(self.call_args["encoding"], self.call_args["decode_errors"]) elif IS_PY3: return "" else: @@ -898,6 +943,7 @@ def __eq__(self, other): return unicode(self) == unicode(other) + __hash__ = None # Avoid DeprecationWarning in Python < 3 def __contains__(self, item): @@ -943,7 +989,6 @@ return int(str(self).strip()) - def output_redirect_is_filename(out): return isinstance(out, basestring) @@ -955,22 +1000,22 @@ return tl._prepend_stack -def special_kwarg_validator(kwargs, invalid_list): - s1 = set(kwargs.keys()) +def special_kwarg_validator(passed_kwargs, merged_kwargs, invalid_list): + s1 = set(passed_kwargs.keys()) invalid_args = [] - for args in invalid_list: + for elem in invalid_list: - if callable(args): - fn = args - ret = fn(kwargs) + if callable(elem): + fn = elem + ret = fn(passed_kwargs, merged_kwargs) invalid_args.extend(ret) else: - args, error_msg = args + elem, error_msg = elem - if s1.issuperset(args): - invalid_args.append((args, error_msg)) + if s1.issuperset(elem): + invalid_args.append((elem, error_msg)) return invalid_args @@ -988,20 +1033,25 @@ fileno = fileno_meth() except UnsupportedOperation: pass - elif isinstance(ob, (int,long)) and ob >= 0: + elif isinstance(ob, (int, long)) and ob >= 0: fileno = ob return fileno +def ob_is_fd_based(ob): + return get_fileno(ob) is not None + + def ob_is_tty(ob): """ checks if an object (like a file-like object) is a tty. """ fileno = get_fileno(ob) is_tty = False - if fileno: + if fileno is not None: is_tty = os.isatty(fileno) return is_tty + def ob_is_pipe(ob): fileno = get_fileno(ob) is_pipe = False @@ -1011,34 +1061,58 @@ return is_pipe -def tty_in_validator(kwargs): +def tty_in_validator(passed_kwargs, merged_kwargs): + # here we'll validate that people aren't randomly shotgun-debugging different tty options and hoping that they'll + # work, without understanding what they do pairs = (("tty_in", "in"), ("tty_out", "out")) invalid = [] - for tty, std in pairs: - if tty in kwargs and ob_is_tty(kwargs.get(std, None)): - args = (tty, std) - error = "`_%s` is a TTY already, so so it doesn't make sense \ -to set up a TTY with `_%s`" % (std, tty) - invalid.append((args, error)) + for tty_type, std in pairs: + if tty_type in passed_kwargs and ob_is_tty(passed_kwargs.get(std, None)): + error = "`_%s` is a TTY already, so so it doesn't make sense to set up a TTY with `_%s`" % (std, tty_type) + invalid.append(((tty_type, std), error)) + + # if unify_ttys is set, then both tty_in and tty_out must both be True + if merged_kwargs["unify_ttys"] and not (merged_kwargs["tty_in"] and merged_kwargs["tty_out"]): + invalid.append(( + ("unify_ttys", "tty_in", "tty_out"), + "`_tty_in` and `_tty_out` must both be True if `_unify_ttys` is True" + )) + + return invalid + + +def fg_validator(passed_kwargs, merged_kwargs): + """ fg is not valid with basically every other option """ + + invalid = [] + msg = """\ +_fg is invalid with nearly every other option, see warning and workaround here: + https://amoffat.github.io/sh/sections/special_arguments.html#fg""" + whitelist = set(("env", "fg", "cwd")) + offending = set(passed_kwargs.keys()) - whitelist + + if "fg" in passed_kwargs and passed_kwargs["fg"] and offending: + invalid.append(("fg", msg)) return invalid -def bufsize_validator(kwargs): + +def bufsize_validator(passed_kwargs, merged_kwargs): """ a validator to prevent a user from saying that they want custom - buffering when they're using an in/out object that will be os.dup'd to the + buffering when they're using an in/out object that will be os.dup'ed to the process, and has its own buffering. an example is a pipe or a tty. it doesn't make sense to tell them to have a custom buffering, since the os controls this. """ invalid = [] - in_ob = kwargs.get("in", None) - out_ob = kwargs.get("out", None) + in_ob = passed_kwargs.get("in", None) + out_ob = passed_kwargs.get("out", None) - in_buf = kwargs.get("in_bufsize", None) - out_buf = kwargs.get("out_bufsize", None) + in_buf = passed_kwargs.get("in_bufsize", None) + out_buf = passed_kwargs.get("out_bufsize", None) - in_no_buf = ob_is_tty(in_ob) or ob_is_pipe(in_ob) - out_no_buf = ob_is_tty(out_ob) or ob_is_pipe(out_ob) + in_no_buf = ob_is_fd_based(in_ob) + out_no_buf = ob_is_fd_based(out_ob) err = "Can't specify an {target} bufsize if the {target} target is a pipe or TTY" @@ -1051,19 +1125,41 @@ return invalid +def env_validator(passed_kwargs, merged_kwargs): + """ a validator to check that env is a dictionary and that all environment variable + keys and values are strings. Otherwise, we would exit with a confusing exit code 255. """ + invalid = [] + + env = passed_kwargs.get("env", None) + if env is None: + return invalid + + if not isinstance(env, Mapping): + invalid.append(("env", "env must be dict-like. Got {!r}".format(env))) + return invalid + + for k, v in passed_kwargs["env"].items(): + if not isinstance(k, str): + invalid.append(("env", "env key {!r} must be a str".format(k))) + if not isinstance(v, str): + invalid.append(("env", "value {!r} of env key {!r} must be a str".format(v, k))) + + return invalid + + class Command(object): """ represents an un-run system program, like "ls" or "cd". because it represents the program itself (and not a running instance of it), it should hold very little state. in fact, the only state it does hold is baked arguments. - + when a Command object is called, the result that is returned is a RunningCommand object, which represents the Command put into an execution state. """ thread_local = threading.local() _call_args = { - "fg": False, # run command in foreground + "fg": False, # run command in foreground # run a command in the background. commands run in the background # ignore SIGHUP and do not automatically exit when the parent process @@ -1073,11 +1169,11 @@ # automatically report exceptions for background commands "bg_exc": True, - "with": False, # prepend the command to every command after it + "with": False, # prepend the command to every command after it "in": None, - "out": None, # redirect STDOUT - "err": None, # redirect STDERR - "err_to_out": None, # redirect STDERR to STDOUT + "out": None, # redirect STDOUT + "err": None, # redirect STDERR + "err_to_out": None, # redirect STDERR to STDOUT # stdin buffer size # 1 for line, 0 for unbuffered, any other number for that amount @@ -1101,6 +1197,8 @@ "piped": None, "iter": None, "iter_noblock": None, + # the amount of time to sleep between polling for the iter output queue + "iter_poll_time": 0.1, "ok_code": 0, "cwd": None, @@ -1117,6 +1215,7 @@ # ssh is one of those programs "tty_in": False, "tty_out": True, + "unify_ttys": False, "encoding": DEFAULT_ENCODING, "decode_errors": "strict", @@ -1167,24 +1266,29 @@ # a callable that produces a log message from an argument tuple of the # command and the args "log_msg": None, + + # whether or not to close all inherited fds. typically, this should be True, as inheriting fds can be a security + # vulnerability + "close_fds": True, + + # a whitelist of the integer fds to pass through to the child process. setting this forces close_fds to be True + "pass_fds": set(), } # this is a collection of validators to make sure the special kwargs make # sense _kwarg_validators = ( - (("fg", "bg"), "Command can't be run in the foreground and background"), - (("fg", "err_to_out"), "Can't redirect STDERR in foreground mode"), (("err", "err_to_out"), "Stderr is already being redirected"), (("piped", "iter"), "You cannot iterate when this command is being piped"), - (("piped", "no_pipe"), "Using a pipe doesn't make sense if you've \ -disabled the pipe"), - (("no_out", "iter"), "You cannot iterate over output if there is no \ -output"), + (("piped", "no_pipe"), "Using a pipe doesn't make sense if you've disabled the pipe"), + (("no_out", "iter"), "You cannot iterate over output if there is no output"), + (("close_fds", "pass_fds"), "Passing `pass_fds` forces `close_fds` to be True"), tty_in_validator, bufsize_validator, + env_validator, + fg_validator, ) - def __init__(self, path, search_paths=None): found = which(path, search_paths) @@ -1206,20 +1310,19 @@ # exception. if CommandNotFound is raised, we need self._path and the # other attributes to be set correctly, so repr() works when they're # inspecting the stack. issue #304 - self._path = encode_to_py3bytes_or_py2str(found) + self._path = encode_to_py3bytes_or_py2str(found) self.__name__ = str(self) - def __getattribute__(self, name): # convenience - getattr = partial(object.__getattribute__, self) + get_attr = partial(object.__getattribute__, self) val = None if name.startswith("_"): - val = getattr(name) + val = get_attr(name) elif name == "bake": - val = getattr("bake") + val = get_attr("bake") # here we have a way of getting past shadowed subcommands. for example, # if "git bake" was a thing, we wouldn't be able to do `git.bake()` @@ -1228,16 +1331,15 @@ name = name[:-1] if val is None: - val = getattr("bake")(name) + val = get_attr("bake")(name) return val - @staticmethod def _extract_call_args(kwargs): """ takes kwargs that were passed to a command's __call__ and extracts out the special keyword arguments, we return a tuple of special keyword - args, and kwargs that will go to the execd command """ + args, and kwargs that will go to the exec'ed command """ kwargs = kwargs.copy() call_args = {} @@ -1248,19 +1350,19 @@ call_args[parg] = kwargs[key] del kwargs[key] - invalid_kwargs = special_kwarg_validator(call_args, - Command._kwarg_validators) + merged_args = Command._call_args.copy() + merged_args.update(call_args) + invalid_kwargs = special_kwarg_validator(call_args, merged_args, Command._kwarg_validators) if invalid_kwargs: exc_msg = [] - for args, error_msg in invalid_kwargs: - exc_msg.append(" %r: %s" % (args, error_msg)) + for kwarg, error_msg in invalid_kwargs: + exc_msg.append(" %r: %s" % (kwarg, error_msg)) exc_msg = "\n".join(exc_msg) raise TypeError("Invalid special arguments:\n\n%s\n" % exc_msg) return call_args, kwargs - # TODO needs documentation def bake(self, *args, **kwargs): fn = type(self)(self._path) @@ -1280,8 +1382,7 @@ fn._partial_call_args.update(pruned_call_args) fn._partial_baked_args.extend(self._partial_baked_args) sep = pruned_call_args.get("long_sep", self._call_args["long_sep"]) - prefix = pruned_call_args.get("long_prefix", - self._call_args["long_prefix"]) + prefix = pruned_call_args.get("long_prefix", self._call_args["long_prefix"]) fn._partial_baked_args.extend(compile_args(args, kwargs, sep, prefix)) return fn @@ -1293,19 +1394,16 @@ else: return self.__unicode__().encode(DEFAULT_ENCODING) - def __eq__(self, other): return str(self) == str(other) __hash__ = None # Avoid DeprecationWarning in Python < 3 - def __repr__(self): """ in python3, should return unicode. in python2, should return a string of bytes """ return "" % str(self) - def __unicode__(self): """ a magic method defined for python2. calling unicode() on a self will call this """ @@ -1317,17 +1415,15 @@ def __enter__(self): self(_with=True) - def __exit__(self, typ, value, traceback): + def __exit__(self, exc_type, exc_val, exc_tb): get_prepend_stack().pop() - def __call__(self, *args, **kwargs): - kwargs = kwargs.copy() args = list(args) # this will hold our final command, including arguments, that will be - # execd + # exec'ed cmd = [] # this will hold a complete mapping of all our special keyword arguments @@ -1358,7 +1454,6 @@ call_args.update(self._partial_call_args) call_args.update(extracted_call_args) - # handle a None. this is added back only to not break the api in the # 1.* version. TODO remove this in 2.0, as "ok_code", if specified, # should always be a definitive value or list of values, and None is @@ -1369,7 +1464,6 @@ if not getattr(call_args["ok_code"], "__iter__", None): call_args["ok_code"] = [call_args["ok_code"]] - # check if we're piping via composition stdin = call_args["in"] if args: @@ -1383,8 +1477,7 @@ else: args.insert(0, first_arg) - processed_args = compile_args(args, kwargs, call_args["long_sep"], - call_args["long_prefix"]) + processed_args = compile_args(args, kwargs, call_args["long_sep"], call_args["long_prefix"]) # makes sure our arguments are broken up correctly split_args = self._partial_baked_args + processed_args @@ -1396,14 +1489,15 @@ # if we're running in foreground mode, we need to completely bypass # launching a RunningCommand and OProc and just do a spawn if call_args["fg"]: - if call_args["env"] is None: - launch = lambda: os.spawnv(os.P_WAIT, cmd[0], cmd) - else: - launch = lambda: os.spawnve(os.P_WAIT, cmd[0], cmd, call_args["env"]) - exit_code = launch() - exc_class = get_exc_exit_code_would_raise(exit_code, - call_args["ok_code"], call_args["piped"]) + cwd = call_args["cwd"] or os.getcwd() + with pushd(cwd): + if call_args["env"] is None: + exit_code = os.spawnv(os.P_WAIT, cmd[0], cmd) + else: + exit_code = os.spawnve(os.P_WAIT, cmd[0], cmd, call_args["env"]) + + exc_class = get_exc_exit_code_would_raise(exit_code, call_args["ok_code"], call_args["piped"]) if exc_class: if IS_PY3: ran = " ".join([arg.decode(DEFAULT_ENCODING, "ignore") for arg in cmd]) @@ -1413,7 +1507,6 @@ raise exc return None - # stdout redirection stdout = call_args["out"] if output_redirect_is_filename(stdout): @@ -1423,11 +1516,11 @@ stderr = call_args["err"] if output_redirect_is_filename(stderr): stderr = open(str(stderr), "wb") - + return RunningCommand(cmd, call_args, stdin, stdout, stderr) -def compile_args(args, kwargs, sep, prefix): +def compile_args(a, kwargs, sep, prefix): """ takes args and kwargs, as they were passed into the command instance being executed with __call__, and compose them into a flat list that will eventually be fed into exec. example: @@ -1444,13 +1537,13 @@ and produces ['-l', '/tmp', '--color=never'] - + """ processed_args = [] encode = encode_to_py3bytes_or_py2str # aggregate positional args - for arg in args: + for arg in a: if isinstance(arg, (list, tuple)): if isinstance(arg, GlobResults) and not arg: arg = [arg.path] @@ -1459,6 +1552,10 @@ processed_args.append(encode(sub_arg)) elif isinstance(arg, dict): processed_args += aggregate_keywords(arg, sep, prefix, raw=True) + + # see https://github.com/amoffat/sh/issues/522 + elif arg is None or arg is False: + pass else: processed_args.append(encode(arg)) @@ -1497,7 +1594,7 @@ ['--some-option=12'] - eessentially, using kwargs is a convenience, but it lacks the ability to + essentially, using kwargs is a convenience, but it lacks the ability to put a '-' in the name, so we do the replacement of '_' to '-' for you. but when you really don't want that to happen, you should use a dictionary instead with the exact names you want @@ -1521,7 +1618,7 @@ k = k.replace("_", "-") if v is True: - processed.append(encode("--" + k)) + processed.append(encode(prefix + k)) elif v is False: pass elif sep is None or sep == " ": @@ -1534,28 +1631,29 @@ return processed -def _start_daemon_thread(fn, name, exc_queue, *args): - def wrap(*args, **kwargs): +def _start_daemon_thread(fn, name, exc_queue, *a): + def wrap(*rgs, **kwargs): try: - fn(*args, **kwargs) + fn(*rgs, **kwargs) except Exception as e: exc_queue.put(e) raise - thrd = threading.Thread(target=wrap, name=name, args=args) - thrd.daemon = True - thrd.start() - return thrd + thread = threading.Thread(target=wrap, name=name, args=a) + thread.daemon = True + thread.start() + return thread def setwinsize(fd, rows_cols): """ set the terminal size of a tty file descriptor. borrowed logic from pexpect.py """ rows, cols = rows_cols - TIOCSWINSZ = getattr(termios, 'TIOCSWINSZ', -2146929561) + winsize = getattr(termios, 'TIOCSWINSZ', -2146929561) s = struct.pack('HHHH', rows, cols, 0, 0) - fcntl.ioctl(fd, TIOCSWINSZ, s) + fcntl.ioctl(fd, winsize, s) + def construct_streamreader_callback(process, handler): """ here we're constructing a closure for our streamreader callback. this @@ -1569,7 +1667,6 @@ or pass some stdin back, and will realize that they can pass a callback of more args """ - # implied arg refers to the "self" that methods will pass in. we need to # account for this implied arg when figuring out what function the user # passed in based on number of args @@ -1595,7 +1692,6 @@ implied_arg = 1 num_args = get_num_args(handler_to_inspect.__call__) - net_args = num_args - implied_arg - partial_args handler_args = () @@ -1618,10 +1714,10 @@ def fn(chunk): # this is pretty ugly, but we're evaluating the process at call-time, # because it's a weakref - args = handler_args - if len(args) == 2: - args = (handler_args[0], handler_args[1]()) - return handler(chunk, *args) + a = handler_args + if len(a) == 2: + a = (handler_args[0], handler_args[1]()) + return handler(chunk, *a) return fn @@ -1687,17 +1783,13 @@ STDOUT = -1 STDERR = -2 - def __init__(self, command, parent_log, cmd, stdin, stdout, stderr, - call_args, pipe, process_assign_lock): + def __init__(self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock): """ - cmd is the full string that will be exec'd. it includes the program - name and all its arguments + cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. - stdin, stdout, stderr are what the child will use for standard - input/output/err + stdin, stdout, stderr are what the child will use for standard input/output/err. - call_args is a mapping of all the special keyword arguments to apply - to the child process + call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args @@ -1713,6 +1805,8 @@ pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid + else: + target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False @@ -1721,37 +1815,46 @@ self._stdin_process = None - # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints - stdin_is_tty_or_pipe = ob_is_tty(stdin) or ob_is_pipe(stdin) - stdout_is_tty_or_pipe = ob_is_tty(stdout) or ob_is_pipe(stdout) - stderr_is_tty_or_pipe = ob_is_tty(stderr) or ob_is_pipe(stderr) + stdin_is_fd_based = ob_is_fd_based(stdin) + stdout_is_fd_based = ob_is_fd_based(stdout) + stderr_is_fd_based = ob_is_fd_based(stderr) tee_out = ca["tee"] in (True, "out") tee_err = ca["tee"] == "err" - # if we're passing in a custom stdout/out/err value, we obviously have - # to force not using single_tty - custom_in_out_err = stdin or stdout or stderr - - single_tty = (ca["tty_in"] and ca["tty_out"])\ - and not custom_in_out_err + single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: - self._stdin_read_fd, self._stdin_write_fd = pty.openpty() - - self._stdout_read_fd = os.dup(self._stdin_read_fd) - self._stdout_write_fd = os.dup(self._stdin_write_fd) + # master_fd, slave_fd = pty.openpty() + # + # Anything that is written on the master end is provided to the process on the slave end as though it was + # input typed on a terminal. -"man 7 pty" + # + # later, in the child process, we're going to do this, so keep it in mind: + # + # os.dup2(self._stdin_child_fd, 0) + # os.dup2(self._stdout_child_fd, 1) + # os.dup2(self._stderr_child_fd, 2) + self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() + + # this makes our parent fds behave like a terminal. it says that the very same fd that we "type" to (for + # stdin) is the same one that we see output printed to (for stdout) + self._stdout_parent_fd = os.dup(self._stdin_parent_fd) + + # this line is what makes stdout and stdin attached to the same pty. in other words the process will write + # to the same underlying fd as stdout as it uses to read from for stdin. this makes programs like ssh happy + self._stdout_child_fd = os.dup(self._stdin_child_fd) - self._stderr_read_fd = os.dup(self._stdin_read_fd) - self._stderr_write_fd = os.dup(self._stdin_write_fd) + self._stderr_parent_fd = os.dup(self._stdin_parent_fd) + self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case @@ -1759,32 +1862,31 @@ # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: - self._stdin_write_fd = stdin._pipe_fd - self._stdin_read_fd = None + self._stdin_child_fd = stdin._pipe_fd + self._stdin_parent_fd = None self._stdin_process = stdin - elif stdin_is_tty_or_pipe: - self._stdin_write_fd = os.dup(get_fileno(stdin)) - self._stdin_read_fd = None + elif stdin_is_fd_based: + self._stdin_child_fd = os.dup(get_fileno(stdin)) + self._stdin_parent_fd = None elif ca["tty_in"]: - self._stdin_read_fd, self._stdin_write_fd = pty.openpty() + self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: - self._stdin_write_fd, self._stdin_read_fd = os.pipe() - + self._stdin_child_fd, self._stdin_parent_fd = os.pipe() - if stdout_is_tty_or_pipe and not tee_out: - self._stdout_write_fd = os.dup(get_fileno(stdout)) - self._stdout_read_fd = None + if stdout_is_fd_based and not tee_out: + self._stdout_child_fd = os.dup(get_fileno(stdout)) + self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: - self._stdout_read_fd, self._stdout_write_fd = pty.openpty() + self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: - self._stdout_read_fd, self._stdout_write_fd = os.pipe() + self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, @@ -1794,50 +1896,36 @@ # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, - # we should not specify a read_fd, because stdout is dup'd + # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup - if stdout_is_tty_or_pipe and not tee_out: - self._stderr_read_fd = None + if stdout_is_fd_based and not tee_out: + self._stderr_parent_fd = None else: - self._stderr_read_fd = os.dup(self._stdout_read_fd) - self._stderr_write_fd = os.dup(self._stdout_write_fd) + self._stderr_parent_fd = os.dup(self._stdout_parent_fd) + self._stderr_child_fd = os.dup(self._stdout_child_fd) - - elif stderr_is_tty_or_pipe and not tee_err: - self._stderr_write_fd = os.dup(get_fileno(stderr)) - self._stderr_read_fd = None + elif stderr_is_fd_based and not tee_err: + self._stderr_child_fd = os.dup(get_fileno(stderr)) + self._stderr_parent_fd = None else: - self._stderr_read_fd, self._stderr_write_fd = os.pipe() - + self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: - fd_to_use = self._stdout_read_fd + fd_to_use = self._stdout_parent_fd if piped == "err": - fd_to_use = self._stderr_read_fd + fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) - new_session = ca["new_session"] needs_ctty = ca["tty_in"] and new_session self.ctty = None if needs_ctty: - self.ctty = os.ttyname(self._stdin_write_fd) - - # this is a hack, but what we're doing here is intentionally throwing an - # OSError exception if our child processes's directory doesn't exist, - # but we're doing it BEFORE we fork. the reason for before the fork is - # error handling. i'm currently too lazy to implement what - # subprocess.py did and set up a error pipe to handle exceptions that - # happen in the child between fork and exec. it has only been seen in - # the wild for a missing cwd, so we'll handle it here. - cwd = ca["cwd"] - if cwd is not None and not os.path.exists(cwd): - os.chdir(cwd) + self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: @@ -1847,13 +1935,15 @@ session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() - # this pipe is for synchronzing with the child that the parent has + # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do - # os.close(self._stdout_write_fd) in the parent after the child starts + # os.close(self._stdout_child_fd) in the parent after the child starts # writing. - if IS_OSX: + if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() + else: + close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None @@ -1861,12 +1951,25 @@ self.pid = os.fork() # child - if self.pid == 0: # pragma: no cover - if IS_OSX: + if self.pid == 0: # pragma: no cover + if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) + # this is critical + # our exc_pipe_write must have CLOEXEC enabled. the reason for this is tricky: + # if our child (the block we're in now), has an exception, we need to be able to write to exc_pipe_write, so + # that when the parent does os.read(exc_pipe_read), it gets our traceback. however, os.read(exc_pipe_read) + # in the parent blocks, so if our child *doesn't* have an exception, and doesn't close the writing end, it + # hangs forever. not good! but obviously the child can't close the writing end until it knows it's not + # going to have an exception, which is impossible to know because but what if os.execv has an exception? so + # the answer is CLOEXEC, so that the writing end of the pipe gets closed upon successful exec, and the + # parent reading the read end won't block (close breaks the block). + flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) + flags |= fcntl.FD_CLOEXEC + fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) + try: # ignoring SIGHUP lets us persist even after the parent process # exits. only ignore if we're backgrounded @@ -1900,7 +2003,7 @@ payload = ("%d,%d" % (sid, pgid)).encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) - if ca["tty_out"] and not stdout_is_tty_or_pipe and not single_tty: + if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways @@ -1908,30 +2011,29 @@ # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. - tty.setraw(self._stdout_write_fd) - + tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping - if self._stdin_read_fd: - os.close(self._stdin_read_fd) + if self._stdin_parent_fd: + os.close(self._stdin_parent_fd) - if self._stdout_read_fd: - os.close(self._stdout_read_fd) + if self._stdout_parent_fd: + os.close(self._stdout_parent_fd) - if self._stderr_read_fd: - os.close(self._stderr_read_fd) + if self._stderr_parent_fd: + os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) + cwd = ca["cwd"] if cwd: os.chdir(cwd) - os.dup2(self._stdin_write_fd, 0) - os.dup2(self._stdout_write_fd, 1) - os.dup2(self._stderr_write_fd, 2) - + os.dup2(self._stdin_child_fd, 0) + os.dup2(self._stdout_child_fd, 1) + os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise @@ -1939,7 +2041,7 @@ tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) - if ca["tty_out"] and not stdout_is_tty_or_pipe: + if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: @@ -1950,10 +2052,22 @@ if callable(preexec_fn): preexec_fn() - - # don't inherit file descriptors - max_fd = resource.getrlimit(resource.RLIMIT_NOFILE)[0] - os.closerange(3, max_fd) + close_fds = ca["close_fds"] + if ca["pass_fds"]: + close_fds = True + + if close_fds: + pass_fds = set((0, 1, 2, exc_pipe_write)) + pass_fds.update(ca["pass_fds"]) + + # don't inherit file descriptors + inherited_fds = os.listdir("/dev/fd") + inherited_fds = set(int(fd) for fd in inherited_fds) - pass_fds + for fd in inherited_fds: + try: + os.close(fd) + except OSError: + pass # actually execute the process if ca["env"] is None: @@ -1968,12 +2082,17 @@ # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. - except: + except: # noqa: E722 # some helpful debugging + tb = traceback.format_exc().encode("utf8", "ignore") + try: - tb = traceback.format_exc().encode("utf8", "ignore") os.write(exc_pipe_write, tb) + except Exception as e: + # dump to stderr if we cannot save it to exc_pipe_write + sys.stderr.write("\nFATAL SH ERROR: %s\n" % e) + finally: os._exit(255) @@ -1982,28 +2101,27 @@ if gc_enabled: gc.enable() - os.close(self._stdin_write_fd) - os.close(self._stdout_write_fd) - os.close(self._stderr_write_fd) + os.close(self._stdin_child_fd) + os.close(self._stdout_child_fd) + os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary - if IS_OSX: + if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) - fork_exc = os.read(exc_pipe_read, 1024**2) + fork_exc = os.read(exc_pipe_read, 1024 ** 2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) raise ForkException(fork_exc) os.close(session_pipe_write) - sid, pgid = os.read(session_pipe_read, - 1024).decode(DEFAULT_ENCODING).split(",") + sid, pgid = os.read(session_pipe_read, 1024).decode(DEFAULT_ENCODING).split(",") os.close(session_pipe_read) self.sid = int(sid) self.pgid = int(pgid) @@ -2020,7 +2138,12 @@ # to prevent race conditions self.exit_code = None - self.stdin = stdin or Queue() + self.stdin = stdin + + # this accounts for when _out is a callable that is passed stdin. in that case, if stdin is unspecified, we + # must set it to a queue, so callbacks can put things on it + if callable(ca["out"]) and self.stdin is None: + self.stdin = Queue() # _pipe_queue is used internally to hand off stdout from one process # to another. by default, all stdout from a process gets dumped @@ -2038,109 +2161,94 @@ self._stdout = deque(maxlen=ca["internal_bufsize"]) self._stderr = deque(maxlen=ca["internal_bufsize"]) - if ca["tty_in"] and not stdin_is_tty_or_pipe: - setwinsize(self._stdin_read_fd, ca["tty_size"]) - + if ca["tty_in"] and not stdin_is_fd_based: + setwinsize(self._stdin_parent_fd, ca["tty_size"]) self.log = parent_log.get_child("process", repr(self)) - self.log.debug("started process") # disable echoing, but only if it's a tty that we created ourselves - if ca["tty_in"] and not stdin_is_tty_or_pipe: - attr = termios.tcgetattr(self._stdin_read_fd) + if ca["tty_in"] and not stdin_is_fd_based: + attr = termios.tcgetattr(self._stdin_parent_fd) attr[3] &= ~termios.ECHO - termios.tcsetattr(self._stdin_read_fd, termios.TCSANOW, attr) - - # we're only going to create a stdin thread iff we have potential - # for stdin to come in. this would be through a stdout callback or - # through an object we've passed in for stdin - potentially_has_input = callable(stdout) or stdin + termios.tcsetattr(self._stdin_parent_fd, termios.TCSANOW, attr) # this represents the connection from a Queue object (or whatever # we're using to feed STDIN) to the process's STDIN fd self._stdin_stream = None - if self._stdin_read_fd and potentially_has_input: + if self._stdin_parent_fd: log = self.log.get_child("streamwriter", "stdin") - self._stdin_stream = StreamWriter(log, self._stdin_read_fd, - self.stdin, ca["in_bufsize"], ca["encoding"], - ca["tty_in"]) + self._stdin_stream = StreamWriter(log, self._stdin_parent_fd, self.stdin, + ca["in_bufsize"], ca["encoding"], ca["tty_in"]) stdout_pipe = None if pipe is OProc.STDOUT and not ca["no_pipe"]: stdout_pipe = self._pipe_queue - # this represents the connection from a process's STDOUT fd to # wherever it has to go, sometimes a pipe Queue (that we will use # to pipe data to other processes), and also an internal deque # that we use to aggregate all the output - save_stdout = not ca["no_out"] and \ - (tee_out or stdout is None) - + save_stdout = not ca["no_out"] and (tee_out or stdout is None) pipe_out = ca["piped"] in ("out", True) pipe_err = ca["piped"] in ("err",) - # if we're piping directly into another process's filedescriptor, we + # if we're piping directly into another process's file descriptor, we # bypass reading from the stdout stream altogether, because we've # already hooked up this processes's stdout fd to the other # processes's stdin fd self._stdout_stream = None - if not pipe_out and self._stdout_read_fd: + if not pipe_out and self._stdout_parent_fd: if callable(stdout): stdout = construct_streamreader_callback(self, stdout) - self._stdout_stream = \ - StreamReader( - self.log.get_child("streamreader", "stdout"), - self._stdout_read_fd, stdout, self._stdout, - ca["out_bufsize"], ca["encoding"], - ca["decode_errors"], stdout_pipe, - save_data=save_stdout) - - elif self._stdout_read_fd: - os.close(self._stdout_read_fd) + self._stdout_stream = StreamReader( + self.log.get_child("streamreader", "stdout"), + self._stdout_parent_fd, stdout, self._stdout, + ca["out_bufsize"], ca["encoding"], + ca["decode_errors"], stdout_pipe, + save_data=save_stdout + ) + elif self._stdout_parent_fd: + os.close(self._stdout_parent_fd) # if stderr is going to one place (because it's grouped with stdout, # or we're dealing with a single tty), then we don't actually need a # stream reader for stderr, because we've already set one up for # stdout above self._stderr_stream = None - if stderr is not OProc.STDOUT and not single_tty and not pipe_err \ - and self._stderr_read_fd: + if stderr is not OProc.STDOUT and not single_tty and not pipe_err and self._stderr_parent_fd: stderr_pipe = None if pipe is OProc.STDERR and not ca["no_pipe"]: stderr_pipe = self._pipe_queue - save_stderr = not ca["no_err"] and \ - (ca["tee"] in ("err",) or stderr is None) + save_stderr = not ca["no_err"] and (ca["tee"] in ("err",) or stderr is None) if callable(stderr): stderr = construct_streamreader_callback(self, stderr) - self._stderr_stream = StreamReader(Logger("streamreader"), - self._stderr_read_fd, stderr, self._stderr, - ca["err_bufsize"], ca["encoding"], ca["decode_errors"], - stderr_pipe, save_data=save_stderr) - - elif self._stderr_read_fd: - os.close(self._stderr_read_fd) + self._stderr_stream = StreamReader( + Logger("streamreader"), + self._stderr_parent_fd, stderr, self._stderr, + ca["err_bufsize"], ca["encoding"], ca["decode_errors"], + stderr_pipe, save_data=save_stderr + ) + elif self._stderr_parent_fd: + os.close(self._stderr_parent_fd) def timeout_fn(): self.timed_out = True self.signal(ca["timeout_signal"]) - self._timeout_event = None self._timeout_timer = None if ca["timeout"]: self._timeout_event = threading.Event() - self._timeout_timer = threading.Timer(ca["timeout"], - self._timeout_event.set) + self._timeout_timer = threading.Timer(ca["timeout"], self._timeout_event.set) self._timeout_timer.start() # this is for cases where we know that the RunningCommand that was @@ -2155,17 +2263,19 @@ def fn(exit_code): with process_assign_lock: return self.command.handle_command_exit_code(exit_code) + handle_exit_code = fn self._quit_threads = threading.Event() thread_name = "background thread for pid %d" % self.pid self._bg_thread_exc_queue = Queue(1) - self._background_thread = _start_daemon_thread(background_thread, - thread_name, self._bg_thread_exc_queue, timeout_fn, - self._timeout_event, handle_exit_code, self.is_alive, - self._quit_threads) - + self._background_thread = _start_daemon_thread( + background_thread, + thread_name, self._bg_thread_exc_queue, timeout_fn, + self._timeout_event, handle_exit_code, self.is_alive, + self._quit_threads + ) # start the main io threads. stdin thread is not needed if we are # connecting from another process's stdout pipe @@ -2174,14 +2284,15 @@ if self._stdin_stream: close_before_term = not needs_ctty thread_name = "STDIN thread for pid %d" % self.pid - self._input_thread = _start_daemon_thread(input_thread, - thread_name, self._input_thread_exc_queue, self.log, - self._stdin_stream, self.is_alive, self._quit_threads, - close_before_term) - + self._input_thread = _start_daemon_thread( + input_thread, + thread_name, self._input_thread_exc_queue, self.log, + self._stdin_stream, self.is_alive, self._quit_threads, + close_before_term + ) # this event is for cases where the subprocess that we launch - # launches its OWN subprocess and dups the stdout/stderr fds to that + # launches its OWN subprocess and os.dup's the stdout/stderr fds to that # new subprocess. in that case, stdout and stderr will never EOF, # so our output_thread will never finish and will hang. this event # prevents that hanging @@ -2189,17 +2300,17 @@ self._output_thread_exc_queue = Queue(1) thread_name = "STDOUT/ERR thread for pid %d" % self.pid - self._output_thread = _start_daemon_thread(output_thread, - thread_name, self._output_thread_exc_queue, self.log, - self._stdout_stream, self._stderr_stream, - self._timeout_event, self.is_alive, self._quit_threads, - self._stop_output_event) - + self._output_thread = _start_daemon_thread( + output_thread, + thread_name, self._output_thread_exc_queue, self.log, + self._stdout_stream, self._stderr_stream, + self._timeout_event, self.is_alive, self._quit_threads, + self._stop_output_event + ) def __repr__(self): return "" % (self.pid, self.cmd[:500]) - # these next 3 properties are primary for tests @property def output_thread_exc(self): @@ -2228,7 +2339,6 @@ pass return exc - def change_in_bufsize(self, buf): self._stdin_stream.stream_bufferer.change_buffering(buf) @@ -2238,8 +2348,6 @@ def change_err_bufsize(self, buf): self._stderr_stream.stream_bufferer.change_buffering(buf) - - @property def stdout(self): return "".encode(self.call_args["encoding"]).join(self._stdout) @@ -2250,13 +2358,13 @@ def get_pgid(self): """ return the CURRENT group id of the process. this differs from - self.pgid in that this refects the current state of the process, where + self.pgid in that this reflects the current state of the process, where self.pgid is the group id at launch """ return os.getpgid(self.pid) def get_sid(self): """ return the CURRENT session id of the process. this differs from - self.sid in that this refects the current state of the process, where + self.sid in that this reflects the current state of the process, where self.sid is the session id at launch """ return os.getsid(self.pid) @@ -2280,7 +2388,6 @@ self.log.debug("terminating") self.signal(signal.SIGTERM) - def is_alive(self): """ polls if our child process has completed, without blocking. this method has side-effects, such as setting our exit_code, if we happen to @@ -2325,7 +2432,6 @@ finally: self._wait_lock.release() - def _process_just_ended(self): if self._timeout_timer: self._timeout_timer.cancel() @@ -2339,9 +2445,8 @@ # the CTTY, and closing it prematurely will send a SIGHUP. we also # don't want to close it if there's a self._stdin_stream, because that # is in charge of closing it also - if self._stdin_read_fd and not self._stdin_stream: - os.close(self._stdin_read_fd) - + if self._stdin_parent_fd and not self._stdin_stream: + os.close(self._stdin_parent_fd) def wait(self): """ waits for the process to complete, handles the exit code """ @@ -2355,13 +2460,12 @@ if self.exit_code is None: self.log.debug("exit code not set, waiting on pid") - pid, exit_code = no_interrupt(os.waitpid, self.pid, 0) # blocks + pid, exit_code = no_interrupt(os.waitpid, self.pid, 0) # blocks self.exit_code = handle_process_exit_code(exit_code) witnessed_end = True else: - self.log.debug("exit code already set (%d), no need to wait", - self.exit_code) + self.log.debug("exit code already set (%d), no need to wait", self.exit_code) self._quit_threads.set() @@ -2389,13 +2493,11 @@ return self.exit_code - -def input_thread(log, stdin, is_alive, quit, close_before_term): +def input_thread(log, stdin, is_alive, quit_thread, close_before_term): """ this is run in a separate thread. it writes into our process's stdin (a streamwriter) and waits the process to end AND everything that can be written to be written """ - done = False closed = False alive = True poller = Poller() @@ -2417,7 +2519,7 @@ alive, _ = is_alive() while alive: - quit.wait(1) + quit_thread.wait(1) alive, _ = is_alive() if not closed: @@ -2431,13 +2533,12 @@ return triggered -def background_thread(timeout_fn, timeout_event, handle_exit_code, is_alive, - quit): +def background_thread(timeout_fn, timeout_event, handle_exit_code, is_alive, quit_thread): """ handles the timeout logic """ - # if there's a timeout event, loop + # if there's a timeout event, loop if timeout_event: - while not quit.is_set(): + while not quit_thread.is_set(): timed_out = event_wait(timeout_event, 0.1) if timed_out: timeout_fn() @@ -2450,17 +2551,17 @@ # this reports the exit code exception in our thread. it's purely for the # user's awareness, and cannot be caught or used in any way, so it's ok to # suppress this during the tests - if handle_exit_code and not RUNNING_TESTS: # pragma: no cover + if handle_exit_code and not RUNNING_TESTS: # pragma: no cover alive = True + exit_code = None while alive: - quit.wait(1) + quit_thread.wait(1) alive, exit_code = is_alive() handle_exit_code(exit_code) -def output_thread(log, stdout, stderr, timeout_event, is_alive, quit, - stop_output_event): +def output_thread(log, stdout, stderr, timeout_event, is_alive, quit_thread, stop_output_event): """ this function is run in a separate thread. it reads from the process's stdout stream (a streamreader), and waits for it to claim that its done """ @@ -2500,7 +2601,7 @@ # outputs, otherwise SIGPIPE alive, _ = is_alive() while alive: - quit.wait(1) + quit_thread.wait(1) alive, _ = is_alive() if stdout: @@ -2510,14 +2611,18 @@ stderr.close() -class DoneReadingForever(Exception): pass -class NotYetReadyToRead(Exception): pass +class DoneReadingForever(Exception): + pass + + +class NotYetReadyToRead(Exception): + pass def determine_how_to_read_input(input_obj): """ given some kind of input object, return a function that knows how to read chunks of that input object. - + each reader function should return a chunk and raise a DoneReadingForever exception, or return None, when there's no more data to read @@ -2526,8 +2631,6 @@ will take care of that. these functions just need to return a reasonably-sized chunk of data. """ - get_chunk = None - if isinstance(input_obj, Queue): log_msg = "queue" get_chunk = get_queue_chunk_reader(input_obj) @@ -2553,6 +2656,14 @@ log_msg = "generator" get_chunk = get_iter_chunk_reader(iter(input_obj)) + elif input_obj is None: + log_msg = "None" + + def raise_(): + raise DoneReadingForever + + get_chunk = raise_ + else: try: it = iter(input_obj) @@ -2565,7 +2676,6 @@ return get_chunk, log_msg - def get_queue_chunk_reader(stdin): def fn(): try: @@ -2575,6 +2685,7 @@ if chunk is None: raise DoneReadingForever return chunk + return fn @@ -2613,8 +2724,10 @@ return chunk except StopIteration: raise DoneReadingForever + return fn + def get_file_chunk_reader(stdin): bufsize = 1024 @@ -2670,7 +2783,6 @@ return bufsize - class StreamWriter(object): """ StreamWriter reads from some input (the stdin param) and writes to a fd (the stream param). the stdin may be a Queue, a callable, something with @@ -2689,14 +2801,11 @@ self.get_chunk, log_msg = determine_how_to_read_input(stdin) self.log.debug("parsed stdin as a %s", log_msg) - def fileno(self): """ defining this allows us to do poll on an instance of this class """ return self.stream - - def write(self): """ attempt to get a chunk of data to write to our child process's stdin, then write it. the return value answers the questions "are we @@ -2717,7 +2826,7 @@ # EOF time try: char = termios.tcgetattr(self.stream)[6][termios.VEOF] - except: + except: # noqa: E722 char = chr(4).encode() # normally, one EOF should be enough to signal to an program @@ -2745,12 +2854,11 @@ return False # if we're not bytes, make us bytes - if IS_PY3 and hasattr(chunk, "encode"): + if IS_PY3 and not isinstance(chunk, bytes): chunk = chunk.encode(self.encoding) for proc_chunk in self.stream_bufferer.process(chunk): - self.log.debug("got chunk size %d: %r", len(proc_chunk), - proc_chunk[:30]) + self.log.debug("got chunk size %d: %r", len(proc_chunk), proc_chunk[:30]) self.log.debug("writing chunk to process") try: @@ -2759,7 +2867,6 @@ self.log.debug("OSError writing stdin chunk") return True - def close(self): self.log.debug("closing, but flushing first") chunk = self.stream_bufferer.flush() @@ -2776,8 +2883,7 @@ def determine_how_to_feed_output(handler, encoding, decode_errors): if callable(handler): - process, finish = get_callback_chunk_consumer(handler, encoding, - decode_errors) + process, finish = get_callback_chunk_consumer(handler, encoding, decode_errors) # in py3, this is used for bytes elif isinstance(handler, (cStringIO, iocStringIO)): @@ -2785,8 +2891,7 @@ # in py3, this is used for unicode elif isinstance(handler, (StringIO, ioStringIO)): - process, finish = get_stringio_chunk_consumer(handler, encoding, - decode_errors) + process, finish = get_stringio_chunk_consumer(handler, encoding, decode_errors) elif hasattr(handler, "write"): process, finish = get_file_chunk_consumer(handler) @@ -2795,8 +2900,8 @@ try: handler = int(handler) except (ValueError, TypeError): - process = lambda chunk: False - finish = lambda: None + def process(chunk): return False # noqa: E731 + def finish(): return None # noqa: E731 else: process, finish = get_fd_chunk_consumer(handler) @@ -2807,14 +2912,17 @@ handler = fdopen(handler, "w", closefd=False) return get_file_chunk_consumer(handler) + def get_file_chunk_consumer(handler): - encode = lambda chunk: chunk if getattr(handler, "encoding", None): - encode = lambda chunk: chunk.decode(handler.encoding) + def encode(chunk): return chunk.decode(handler.encoding) # noqa: E731 + else: + def encode(chunk): return chunk # noqa: E731 - flush = lambda: None if hasattr(handler, "flush"): flush = handler.flush + else: + def flush(): return None # noqa: E731 def process(chunk): handler.write(encode(chunk)) @@ -2828,6 +2936,7 @@ return process, finish + def get_callback_chunk_consumer(handler, encoding, decode_errors): def process(chunk): # try to use the encoding first, if that doesn't work, send @@ -2843,6 +2952,7 @@ return process, finish + def get_cstringio_chunk_consumer(handler): def process(chunk): handler.write(chunk) @@ -2868,8 +2978,9 @@ class StreamReader(object): """ reads from some output (the stream) and sends what it just read to the handler. """ - def __init__(self, log, stream, handler, buffer, bufsize_type, encoding, - decode_errors, pipe_queue=None, save_data=True): + + def __init__(self, log, stream, handler, buffer, bufsize_type, encoding, decode_errors, pipe_queue=None, + save_data=True): self.stream = stream self.buffer = buffer self.save_data = save_data @@ -2882,16 +2993,14 @@ self.log = log - self.stream_bufferer = StreamBufferer(bufsize_type, self.encoding, - self.decode_errors) + self.stream_bufferer = StreamBufferer(bufsize_type, self.encoding, self.decode_errors) self.bufsize = bufsize_type_to_bufsize(bufsize_type) self.process_chunk, self.finish_chunk_processor = \ - determine_how_to_feed_output(handler, encoding, decode_errors) + determine_how_to_feed_output(handler, encoding, decode_errors) self.should_quit = False - def fileno(self): """ defining this allows us to do poll on an instance of this class """ @@ -2910,14 +3019,12 @@ os.close(self.stream) - def write_chunk(self, chunk): # in PY3, the chunk coming in will be bytes, so keep that in mind if not self.should_quit: self.should_quit = self.process_chunk(chunk) - if self.save_data: self.buffer.append(chunk) @@ -2925,7 +3032,6 @@ self.log.debug("putting chunk onto pipe: %r", chunk[:30]) self.pipe_queue().put(chunk) - def read(self): # if we're PY3, we're reading bytes, otherwise we're reading # str @@ -2943,8 +3049,6 @@ self.write_chunk(chunk) - - class StreamBufferer(object): """ this is used for feeding in chunks of stdout/stderr, and breaking it up into chunks that will actually be put into the internal buffers. for @@ -2953,8 +3057,7 @@ however they come in), OProc will use an instance of this class to chop up the data and feed it as lines to be sent down the pipe """ - def __init__(self, buffer_type, encoding=DEFAULT_ENCODING, - decode_errors="strict"): + def __init__(self, buffer_type, encoding=DEFAULT_ENCODING, decode_errors="strict"): # 0 for unbuffered, 1 for line, everything else for that amount self.type = buffer_type self.buffer = [] @@ -2975,7 +3078,6 @@ self._buffering_lock = threading.RLock() self.log = Logger("stream_bufferer") - def change_buffering(self, new_type): # TODO, when we stop supporting 2.6, make this a with context self.log.debug("acquiring buffering lock for changing buffering") @@ -2990,7 +3092,6 @@ self._buffering_lock.release() self.log.debug("released buffering lock for changing buffering") - def process(self, chunk): # MAKE SURE THAT THE INPUT IS PY3 BYTES # THE OUTPUT IS ALWAYS PY3 BYTES @@ -3056,7 +3157,6 @@ self._buffering_lock.release() self.log.debug("released buffering lock for processing chunk (buffering: %d)", self.type) - def flush(self): self.log.debug("acquiring buffering lock for flushing buffer") self._buffering_lock.acquire() @@ -3070,16 +3170,18 @@ self.log.debug("released buffering lock for flushing buffer") - def with_lock(lock): def wrapped(fn): fn = contextmanager(fn) + @contextmanager def wrapped2(*args, **kwargs): with lock: with fn(*args, **kwargs): yield + return wrapped2 + return wrapped @@ -3097,11 +3199,11 @@ @contextmanager -def args(**kwargs): +def _args(**kwargs): """ allows us to temporarily override all the special keyword parameters in a with context """ - kwargs_str = ",".join(["%s=%r" % (k,v) for k,v in kwargs.items()]) + kwargs_str = ",".join(["%s=%r" % (k, v) for k, v in kwargs.items()]) raise DeprecationWarning(""" @@ -3120,7 +3222,6 @@ """.format(kwargs=kwargs_str)) - class Environment(dict): """ this allows lookups to names that aren't found in the global scope to be searched for as a program name. for example, if "ls" isn't found in this @@ -3130,12 +3231,11 @@ exec() statement used in the run_repl requires the "globals" argument to be a dictionary """ - # this is a list of all of the names that the sh module exports that will # not resolve to functions. we don't want to accidentally shadow real # commands with functions/imports that we define in sh.py. for example, # "import time" may override the time system program - whitelist = set([ + whitelist = set(( "Command", "RunningCommand", "CommandNotFound", @@ -3146,71 +3246,62 @@ "SignalException", "ForkException", "TimeoutException", + "StreamBufferer", "__project_url__", "__version__", "__file__", - "args", + "_args", "pushd", "glob", "contrib", - ]) - + )) - def __init__(self, globs, baked_args={}): + def __init__(self, globs, baked_args=None): """ baked_args are defaults for the 'sh' execution context. for example: - + tmp = sh(_out=StringIO()) 'out' would end up in here as an entry in the baked_args dict """ - + super(dict, self).__init__() self.globs = globs - self.baked_args = baked_args - self.disable_whitelist = False + self.baked_args = baked_args or {} def __getitem__(self, k): - # if we first import "_disable_whitelist" from sh, we can import - # anything defined in the global scope of sh.py. this is useful for our - # tests - if k == "_disable_whitelist": - self.disable_whitelist = True - return None - - # we're trying to import something real (maybe), see if it's in our - # global scope - if k in self.whitelist or self.disable_whitelist: + if k == 'args': + # Let the deprecated '_args' context manager be imported as 'args' + k = '_args' + + # if we're trying to import something real, see if it's in our global scope. + # what defines "real" is that it's in our whitelist + if k in self.whitelist: return self.globs[k] # somebody tried to be funny and do "from sh import *" if k == "__all__": - raise RuntimeError("Cannot import * from sh. \ -Please import sh or import programs individually.") - + warnings.warn("Cannot import * from sh. Please import sh or import programs individually.") + return [] # check if we're naming a dynamically generated ReturnCode exception exc = get_exc_from_name(k) if exc: return exc - # https://github.com/ipython/ipython/issues/2577 # https://github.com/amoffat/sh/issues/97#issuecomment-10610629 if k.startswith("__") and k.endswith("__"): raise AttributeError - # is it a custom builtin? builtin = getattr(self, "b_" + k, None) if builtin: return builtin - # is it a command? cmd = resolve_command(k, self.baked_args) if cmd: return cmd - # how about an environment variable? # this check must come after testing if its a command, because on some # systems, there are an environment variables that can conflict with @@ -3221,31 +3312,29 @@ except KeyError: pass - # nothing found, raise an exception raise CommandNotFound(k) - # methods that begin with "b_" are custom builtins and will override any # program that exists in our path. this is useful for things like # common shell builtins that people are used to, but which aren't actually # full-fledged system binaries - - def b_cd(self, path=None): + @staticmethod + def b_cd(path=None): if path: os.chdir(path) else: os.chdir(os.path.expanduser('~')) - def b_which(self, program, paths=None): + @staticmethod + def b_which(program, paths=None): return which(program, paths) -class Contrib(ModuleType): # pragma: no cover +class Contrib(ModuleType): # pragma: no cover @classmethod def __call__(cls, name): def wrapper1(fn): - @property def cmd_getter(self): cmd = resolve_command(name) @@ -3268,13 +3357,14 @@ @contrib("git") -def git(orig): # pragma: no cover +def git(orig): # pragma: no cover """ most git commands play nicer without a TTY """ cmd = orig.bake(_tty_out=False) return cmd + @contrib("sudo") -def sudo(orig): # pragma: no cover +def sudo(orig): # pragma: no cover """ a nicer version of sudo that uses getpass to ask for a password, or allows the first argument to be a string password """ @@ -3284,8 +3374,7 @@ pw = getpass.getpass(prompt=prompt) + "\n" yield pw - - def process(args, kwargs): + def process(a, kwargs): password = kwargs.pop("password", None) if password is None: @@ -3294,15 +3383,93 @@ pass_getter = password.rstrip("\n") + "\n" kwargs["_in"] = pass_getter - return args, kwargs + return a, kwargs cmd = orig.bake("-S", _arg_preprocess=process) return cmd +@contrib("ssh") +def ssh(orig): # pragma: no cover + """ An ssh command for automatic password login """ + + class SessionContent(object): + def __init__(self): + self.chars = deque(maxlen=50000) + self.lines = deque(maxlen=5000) + self.line_chars = [] + self.last_line = "" + self.cur_char = "" + + def append_char(self, char): + if char == "\n": + line = self.cur_line + self.last_line = line + self.lines.append(line) + self.line_chars = [] + else: + self.line_chars.append(char) + + self.chars.append(char) + self.cur_char = char + @property + def cur_line(self): + line = "".join(self.line_chars) + return line + + class SSHInteract(object): + def __init__(self, prompt_match, pass_getter, out_handler, login_success): + self.prompt_match = prompt_match + self.pass_getter = pass_getter + self.out_handler = out_handler + self.login_success = login_success + self.content = SessionContent() + + # some basic state + self.pw_entered = False + self.success = False + + def __call__(self, char, stdin): + self.content.append_char(char) + + if self.pw_entered and not self.success: + self.success = self.login_success(self.content) + + if self.success: + return self.out_handler(self.content, stdin) + + if self.prompt_match(self.content): + password = self.pass_getter() + stdin.put(password + "\n") + self.pw_entered = True + + def process(a, kwargs): + real_out_handler = kwargs.pop("interact") + password = kwargs.pop("password", None) + login_success = kwargs.pop("login_success", None) + prompt_match = kwargs.pop("prompt", None) + prompt = "Please enter SSH password: " -def run_repl(env): # pragma: no cover + if prompt_match is None: + def prompt_match(content): return content.cur_line.endswith("password: ") # noqa: E731 + + if password is None: + def pass_getter(): return getpass.getpass(prompt=prompt) # noqa: E731 + else: + def pass_getter(): return password.rstrip("\n") # noqa: E731 + + if login_success is None: + def login_success(content): return True # noqa: E731 + + kwargs["_out"] = SSHInteract(prompt_match, pass_getter, real_out_handler, login_success) + return a, kwargs + + cmd = orig.bake(_out_bufsize=0, _tty_in=True, _unify_ttys=True, _arg_preprocess=process) + return cmd + + +def run_repl(env): # pragma: no cover banner = "\n>> sh v{version}\n>> https://github.com/amoffat/sh\n" print(banner.format(version=__version__)) @@ -3316,27 +3483,29 @@ exec(compile(line, "", "single"), env, env) except SystemExit: break - except: + except: # noqa: E722 print(traceback.format_exc()) # cleans up our last line print("") - - # this is a thin wrapper around THIS module (we patch sys.modules[__name__]). # this is in the case that the user does a "from sh import whatever" # in other words, they only want to import certain programs, not the whole # system PATH worth of commands. in this case, we just proxy the # import lookup to our Environment class class SelfWrapper(ModuleType): - def __init__(self, self_module, baked_args={}): + def __init__(self, self_module, baked_args=None): # this is super ugly to have to copy attributes like this, # but it seems to be the only way to make reload() behave # nicely. if i make these attributes dynamic lookups in # __getattr__, reload sometimes chokes in weird ways... - for attr in ["__builtins__", "__doc__", "__file__", "__name__", "__package__"]: + super(SelfWrapper, self).__init__( + name=getattr(self_module, '__name__', None), + doc=getattr(self_module, '__doc__', None) + ) + for attr in ["__builtins__", "__file__", "__package__"]: setattr(self, attr, getattr(self_module, attr, None)) # python 3.2 (2.7 and 3.3 work fine) breaks on osx (not ubuntu) @@ -3364,12 +3533,32 @@ # cached module from sys.modules. if we don't, it gets re-used, and any # old baked params get used, which is not what we want parent = inspect.stack()[1] - code = parent[4][0].strip() - parsed = ast.parse(code) - module_name = parsed.body[0].targets[0].id + try: + code = parent[4][0].strip() + except TypeError: + # On the REPL or from the commandline, we don't get the source code in the + # top stack frame + # Older versions of pypy don't set parent[1] the same way as CPython or newer versions + # of Pypy so we have to special case that too. + if parent[1] in ('', '') or ( + parent[1] == '' and platform.python_implementation().lower() == 'pypy'): + # This depends on things like Python's calling convention and the layout of stack + # frames but it's a fix for a bug in a very cornery cornercase so.... + module_name = parent[0].f_code.co_names[-1] + else: + raise + else: + parsed = ast.parse(code) + try: + module_name = parsed.body[0].targets[0].id + except Exception: + # Diagnose what went wrong + if not isinstance(parsed.body[0], ast.Assign): + raise RuntimeError("A new execution context must be assigned to a variable") + raise if module_name == __name__: - raise RuntimeError("Cannot use the name 'sh' as an execution context") + raise RuntimeError("Cannot use the name '%s' as an execution context" % __name__) sys.modules.pop(module_name, None) @@ -3390,35 +3579,39 @@ from tmp import ls """ - def test(importer): - return importer.__class__.__name__ == ModuleImporterFromVariables.__name__ + def test(importer_cls): + try: + return importer_cls.__class__.__name__ == ModuleImporterFromVariables.__name__ + except AttributeError: + # ran into importer which is not a class instance + return False + already_registered = any([True for i in sys.meta_path if test(i)]) if not already_registered: - importer = ModuleImporterFromVariables( - restrict_to=["SelfWrapper"], - ) + importer = ModuleImporterFromVariables(restrict_to=[SelfWrapper.__name__], ) sys.meta_path.insert(0, importer) return not already_registered + def fetch_module_from_frame(name, frame): mod = frame.f_locals.get(name, frame.f_globals.get(name, None)) return mod + class ModuleImporterFromVariables(object): """ a fancy importer that allows us to import from a variable that was recently set in either the local or global scope, like this: sh2 = sh(_timeout=3) from sh2 import ls - + """ def __init__(self, restrict_to=None): self.restrict_to = set(restrict_to or set()) - def find_module(self, mod_fullname, path=None): """ mod_fullname doubles as the name of the VARIABLE holding our new sh context. for example: @@ -3426,15 +3619,26 @@ derp = sh() from derp import ls - here, mod_fullname will be "derp". keep that in mind as we go throug + here, mod_fullname will be "derp". keep that in mind as we go through the rest of this function """ parent_frame = inspect.currentframe().f_back - while in_importlib(parent_frame): + + if parent_frame and parent_frame.f_code.co_name == "find_spec": + parent_frame = parent_frame.f_back + + while parent_frame and in_importlib(parent_frame): parent_frame = parent_frame.f_back + # Calling PyImport_ImportModule("some_module"); via the C API may not + # have a parent frame. Early-out to avoid in_importlib() trying to + # get f_code from None when looking for 'some_module'. + # This also happens when using gevent apparently. + if not parent_frame: + return None + # this line is saying "hey, does mod_fullname exist as a name we've - # defind previously?" the purpose of this is to ensure that + # defined previously?" the purpose of this is to ensure that # mod_fullname is really a thing we've defined. if we haven't defined # it before, then we "can't" import from it module = fetch_module_from_frame(mod_fullname, parent_frame) @@ -3447,6 +3651,12 @@ return self + def find_spec(self, fullname, path=None, target=None): + """ find_module() is deprecated since Python 3.4 in favor of find_spec() """ + + from importlib.machinery import ModuleSpec + found = self.find_module(fullname, path) + return ModuleSpec(fullname, found) if found is not None else None def load_module(self, mod_fullname): parent_frame = inspect.currentframe().f_back @@ -3457,7 +3667,7 @@ module = fetch_module_from_frame(mod_fullname, parent_frame) # we HAVE to include the module in sys.modules, per the import PEP. - # older verions of python were more lenient about this being set, but + # older versions of python were more lenient about this being set, but # not in >= python3.3, unfortunately. this requirement necessitates the # ugly code in SelfWrapper.__call__ sys.modules[mod_fullname] = module @@ -3466,7 +3676,7 @@ return module -def run_tests(env, locale, args, version, force_select, **extra_env): # pragma: no cover +def run_tests(env, locale, a, version, force_select, **extra_env): # pragma: no cover py_version = "python" py_version += str(version) @@ -3478,50 +3688,43 @@ poller = "select" if py_bin: - print("Testing %s, locale %r, poller: %s" % (py_version.capitalize(), - locale, poller)) + print("Testing %s, locale %r, poller: %s" % (py_version.capitalize(), locale, poller)) env["SH_TESTS_USE_SELECT"] = str(int(force_select)) env["LANG"] = locale - for k,v in extra_env.items(): + for k, v in extra_env.items(): env[k] = str(v) - cmd = [py_bin, "-W", "ignore", os.path.join(THIS_DIR, "test.py")] + args[1:] - launch = lambda: os.spawnve(os.P_WAIT, cmd[0], cmd, env) - return_code = launch() + cmd = [py_bin, "-W", "ignore", os.path.join(THIS_DIR, "test.py")] + a[1:] + print("Running %r" % cmd) + return_code = os.spawnve(os.P_WAIT, cmd[0], cmd, env) return return_code +def main(): # pragma: no cover + from optparse import OptionParser -# we're being run as a stand-alone script -if __name__ == "__main__": # pragma: no cover - def parse_args(): - from optparse import OptionParser - - parser = OptionParser() - parser.add_option("-e", "--envs", dest="envs", action="append") - parser.add_option("-l", "--locales", dest="constrain_locales", action="append") - options, args = parser.parse_args() - - envs = options.envs or [] - constrain_locales = options.constrain_locales or [] - - return args, envs, constrain_locales + parser = OptionParser() + parser.add_option("-e", "--envs", dest="envs", default=None, action="append") + parser.add_option("-l", "--locales", dest="constrain_locales", default=None, action="append") + options, parsed_args = parser.parse_args() # these are essentially restrictions on what envs/constrain_locales to restrict to for # the tests. if they're empty lists, it means use all available - args, constrain_versions, constrain_locales = parse_args() action = None - if args: - action = args[0] + if parsed_args: + action = parsed_args[0] - if action in ("test", "travis"): + if action in ("test", "travis", "tox"): import test coverage = None if test.HAS_UNICODE_LITERAL: - import coverage + try: + import coverage + except ImportError: + pass env = os.environ.copy() env["SH_TESTS_RUNNING"] = "1" @@ -3530,12 +3733,11 @@ # if we're testing locally, run all versions of python on the system if action == "test": - all_versions = ("2.6", "2.7", "3.1", "3.2", "3.3", "3.4", "3.5", "3.6") + all_versions = ("2.6", "2.7", "3.1", "3.2", "3.3", "3.4", "3.5", "3.6", "3.7", "3.8") - # if we're testing on travis, just use the system's default python, - # since travis will spawn a vm per python version in our .travis.yml - # file - elif action == "travis": + # if we're testing on travis or tox, just use the system's default python, since travis will spawn a vm per + # python version in our .travis.yml file, and tox will run its matrix via tox.ini + else: v = sys.version_info sys_ver = "%d.%d" % (v[0], v[1]) all_versions = (sys_ver,) @@ -3546,19 +3748,22 @@ all_locales = ("en_US.UTF-8", "C") i = 0 + ran_versions = set() for locale in all_locales: - if constrain_locales and locale not in constrain_locales: + # make sure this locale is allowed + if options.constrain_locales and locale not in options.constrain_locales: continue for version in all_versions: - if constrain_versions and version not in constrain_versions: + # make sure this version is allowed + if options.envs and version not in options.envs: continue for force_select in all_force_select: env_copy = env.copy() - exit_code = run_tests(env_copy, locale, args, version, - force_select, SH_TEST_RUN_IDX=i) + ran_versions.add(version) + exit_code = run_tests(env_copy, locale, parsed_args, version, force_select, SH_TEST_RUN_IDX=i) if exit_code is None: print("Couldn't find %s, skipping" % version) @@ -3569,16 +3774,17 @@ i += 1 - ran_versions = ",".join(all_versions) - print("Tested Python versions: %s" % ran_versions) + print("Tested Python versions: %s" % ",".join(sorted(list(ran_versions)))) else: env = Environment(globals()) run_repl(env) -# we're being imported from somewhere + +if __name__ == "__main__": # pragma: no cover + # we're being run as a stand-alone script + main() else: - self = sys.modules[__name__] - sys.modules[__name__] = SelfWrapper(self) + # we're being imported from somewhere + sys.modules[__name__] = SelfWrapper(sys.modules[__name__]) register_importer() - diff -Nru python-sh-1.12.14/test.py python-sh-1.14.1/test.py --- python-sh-1.12.14/test.py 2018-09-04 13:03:55.000000000 +0000 +++ python-sh-1.14.1/test.py 2020-10-24 17:34:57.000000000 +0000 @@ -1,6 +1,23 @@ # -*- coding: utf8 -*- -import sys +from contextlib import contextmanager +from functools import wraps +from os.path import exists, join, realpath, dirname, split +import errno +import fcntl +import inspect +import logging import os +import platform +import pty +import resource +import sh +import signal +import stat +import sys +import tempfile +import time +import unittest +import warnings IS_PY3 = sys.version_info[0] == 3 IS_PY2 = not IS_PY3 @@ -15,50 +32,35 @@ run_idx = int(os.environ.pop("SH_TEST_RUN_IDX", "0")) first_run = run_idx == 0 - import coverage - - # for some reason, we can't run auto_data on the first run, or the coverage - # numbers get really screwed up - auto_data = True - if first_run: - auto_data = False - - cov = coverage.Coverage(auto_data=auto_data) + try: + import coverage + except ImportError: + pass + else: + # for some reason, we can't run auto_data on the first run, or the coverage + # numbers get really screwed up + auto_data = True + if first_run: + auto_data = False - if first_run: - cov.erase() + cov = coverage.Coverage(auto_data=auto_data) - cov.start() + if first_run: + cov.erase() + cov.start() -from os.path import exists, join, realpath, dirname, split -import unittest try: import unittest.mock except ImportError: HAS_MOCK = False else: HAS_MOCK = True -import tempfile -import warnings -import pty -import resource -import logging -import sys -from contextlib import contextmanager -import sh -import signal -import errno -import stat -import platform -from functools import wraps -import time -import inspect # we have to use the real path because on osx, /tmp is a symlink to # /private/tmp, and so assertions that gettempdir() == sh.pwd() will fail tempdir = realpath(tempfile.gettempdir()) -IS_OSX = platform.system() == "Darwin" +IS_MACOS = platform.system() in ("AIX", "Darwin") # these 3 functions are helpers for modifying PYTHONPATH with a module's main @@ -71,6 +73,7 @@ pypath = ":".join(pypath) env[key] = pypath + def get_module_import_dir(m): mod_file = inspect.getsourcefile(m) is_package = mod_file.endswith("__init__.py") @@ -80,36 +83,36 @@ mod_dir, _ = split(mod_dir) return mod_dir + def append_module_path(env, m): append_pythonpath(env, get_module_import_dir(m)) - if IS_PY3: xrange = range unicode = str long = int from io import StringIO + ioStringIO = StringIO from io import BytesIO as cStringIO + iocStringIO = cStringIO - python = sh.Command(sh.which("python%d.%d" % sys.version_info[:2])) else: from StringIO import StringIO from cStringIO import StringIO as cStringIO from io import StringIO as ioStringIO from io import BytesIO as iocStringIO - python = sh.python THIS_DIR = dirname(os.path.abspath(__file__)) +system_python = sh.Command(sys.executable) # this is to ensure that our `python` helper here is able to import our local sh # module, and not the system one baked_env = os.environ.copy() append_module_path(baked_env, sh) -python = python.bake(_env=baked_env) - +python = system_python.bake(_env=baked_env) if hasattr(logging, 'NullHandler'): NullHandler = logging.NullHandler @@ -124,7 +127,6 @@ def createLock(self): self.lock = None - skipUnless = getattr(unittest, "skipUnless", None) if not skipUnless: # our stupid skipUnless wrapper for python2.6 @@ -136,10 +138,13 @@ @wraps(test) def skip(*args, **kwargs): return + return skip + return wrapper skip_unless = skipUnless + def requires_progs(*progs): missing = [] for prog in progs: @@ -150,11 +155,12 @@ friendly_missing = ", ".join(missing) return skipUnless(len(missing) == 0, "Missing required system programs: %s" - % friendly_missing) + % friendly_missing) + requires_posix = skipUnless(os.name == "posix", "Requires POSIX") requires_utf8 = skipUnless(sh.DEFAULT_ENCODING == "UTF-8", "System encoding must be UTF-8") -not_osx = skipUnless(not IS_OSX, "Doesn't work on OSX") +not_macos = skipUnless(not IS_MACOS, "Doesn't work on MacOS") requires_py3 = skipUnless(IS_PY3, "Test only works on Python 3") requires_py35 = skipUnless(IS_PY3 and MINOR_VER >= 5, "Test only works on Python 3.5 or higher") @@ -209,8 +215,48 @@ with warnings.catch_warnings(record=True) as w: fn(*args, **kwargs) - assert len(w) == 1 - assert issubclass(w[-1].category, DeprecationWarning) + self.assertEqual(len(w), 1) + self.assertTrue(issubclass(w[-1].category, DeprecationWarning)) + + # python2.6 lacks this + def assertIn(self, needle, haystack): + s = super(BaseTests, self) + if hasattr(s, "assertIn"): + s.assertIn(needle, haystack) + else: + self.assertTrue(needle in haystack) + + # python2.6 lacks this + def assertNotIn(self, needle, haystack): + s = super(BaseTests, self) + if hasattr(s, "assertNotIn"): + s.assertNotIn(needle, haystack) + else: + self.assertTrue(needle not in haystack) + + # python2.6 lacks this + def assertLess(self, a, b): + s = super(BaseTests, self) + if hasattr(s, "assertLess"): + s.assertLess(a, b) + else: + self.assertTrue(a < b) + + # python2.6 lacks this + def assertGreater(self, a, b): + s = super(BaseTests, self) + if hasattr(s, "assertGreater"): + s.assertGreater(a, b) + else: + self.assertTrue(a > b) + + # python2.6 lacks this + def skipTest(self, msg): + s = super(BaseTests, self) + if hasattr(s, "skipTest"): + s.skipTest(msg) + else: + return @requires_posix @@ -228,7 +274,6 @@ out = str(ls) self.assertEqual(out, actual_location) - def test_unicode_arg(self): from sh import echo @@ -245,16 +290,14 @@ py = create_tmp_test("exit(1)") arg = "漢字" + native_arg = arg if not IS_PY3: arg = arg.decode("utf8") try: python(py.name, arg, _encoding="utf8") except ErrorReturnCode as e: - if IS_PY3: - self.assertTrue(arg in str(e)) - else: - self.assertTrue(arg in unicode(e)) + self.assertIn(native_arg, str(e)) else: self.fail("exception wasn't raised") @@ -272,7 +315,7 @@ sys.stderr.write("b" * 1000) exit(1) """) - self.assertRaises(sh.ErrorReturnCode, python, py.name) + self.assertRaises(sh.ErrorReturnCode_1, python, py.name) def test_number_arg(self): py = create_tmp_test(""" @@ -285,17 +328,28 @@ out = python(py.name, 3).strip() self.assertEqual(out, "3") + def test_empty_stdin_no_hang(self): + py = create_tmp_test(""" +import sys +data = sys.stdin.read() +sys.stdout.write("no hang") +""") + out = python(py.name, _in="", _timeout=2) + self.assertEqual(out, "no hang") + + out = python(py.name, _in=None, _timeout=2) + self.assertEqual(out, "no hang") def test_exit_code(self): - from sh import ErrorReturnCode + from sh import ErrorReturnCode_3 py = create_tmp_test(""" exit(3) """) - self.assertRaises(ErrorReturnCode, python, py.name) + self.assertRaises(ErrorReturnCode_3, python, py.name) def test_patched_glob(self): from glob import glob - + py = create_tmp_test(""" import sys print(sys.argv[1:]) @@ -317,7 +371,7 @@ self.assertEqual(out, "['*.faowjefoajweofj']") def test_exit_code_with_hasattr(self): - from sh import ErrorReturnCode + from sh import ErrorReturnCode_3 py = create_tmp_test(""" exit(3) """) @@ -329,35 +383,33 @@ list(out) self.assertEqual(out.exit_code, 3) self.fail("Command exited with error, but no exception thrown") - except ErrorReturnCode as e: + except ErrorReturnCode_3: pass - def test_exit_code_from_exception(self): - from sh import ErrorReturnCode + from sh import ErrorReturnCode_3 py = create_tmp_test(""" exit(3) """) - self.assertRaises(ErrorReturnCode, python, py.name) + self.assertRaises(ErrorReturnCode_3, python, py.name) try: python(py.name) except Exception as e: self.assertEqual(e.exit_code, 3) - def test_stdin_from_string(self): from sh import sed self.assertEqual(sed(_in="one test three", e="s/test/two/").strip(), - "one two three") + "one two three") def test_ok_code(self): from sh import ls, ErrorReturnCode_1, ErrorReturnCode_2 exc_to_test = ErrorReturnCode_2 code_to_pass = 2 - if IS_OSX: + if IS_MACOS: exc_to_test = ErrorReturnCode_1 code_to_pass = 1 self.assertRaises(exc_to_test, ls, "/aofwje/garogjao4a/eoan3on") @@ -370,6 +422,24 @@ py = create_tmp_test("exit(0)") python(py.name, _ok_code=None) + def test_ok_code_exception(self): + from sh import ErrorReturnCode_0 + py = create_tmp_test("exit(0)") + self.assertRaises(ErrorReturnCode_0, python, py.name, _ok_code=2) + + def test_none_arg(self): + py = create_tmp_test(""" +import sys +print(sys.argv[1:]) +""") + maybe_arg = "some" + out = python(py.name, maybe_arg).strip() + self.assertEqual(out, "['some']") + + maybe_arg = None + out = python(py.name, maybe_arg).strip() + self.assertEqual(out, "[]") + def test_quote_escaping(self): py = create_tmp_test(""" from optparse import OptionParser @@ -396,7 +466,6 @@ self.assertEqual(out, "[\"one two's three\"]") def test_multiple_pipes(self): - from sh import tr, python import time py = create_tmp_test(""" @@ -437,9 +506,9 @@ derp = Derp() p = inc( + inc( inc( - inc( - python("-u", py.name, _piped=True), + python("-u", py.name, _piped=True), _piped=True), _piped=True), _out=derp.agg) @@ -448,7 +517,6 @@ self.assertEqual("".join(derp.stdout), "dqguhz") self.assertTrue(all([t > .15 for t in derp.times])) - def test_manual_stdin_string(self): from sh import tr @@ -464,7 +532,6 @@ match = "".join([t.upper() for t in test]) self.assertEqual(out, match) - def test_manual_stdin_file(self): from sh import tr import tempfile @@ -480,24 +547,25 @@ self.assertEqual(out, test_string.upper()) - def test_manual_stdin_queue(self): from sh import tr - try: from Queue import Queue, Empty - except ImportError: from queue import Queue, Empty + try: + from Queue import Queue + except ImportError: + from queue import Queue test = ["testing\n", "herp\n", "derp\n"] q = Queue() - for t in test: q.put(t) - q.put(None) # EOF + for t in test: + q.put(t) + q.put(None) # EOF out = tr("[:lower:]", "[:upper:]", _in=q) match = "".join([t.upper() for t in test]) self.assertEqual(out, match) - def test_environment(self): """ tests that environments variables that we pass into sh commands exist in the environment, and on the sh module """ @@ -506,49 +574,41 @@ # this is the environment we'll pass into our commands env = {"HERP": "DERP"} - # python on osx will bizarrely add some extra environment variables that - # i didn't ask for. for this test, we prune those out if they exist - osx_cruft = [ - "__CF_USER_TEXT_ENCODING", - "__PYVENV_LAUNCHER__", - "VERSIONER_PYTHON_PREFER_32_BIT", - "VERSIONER_PYTHON_VERSION", - ] - # first we test that the environment exists in our child process as # we've set it py = create_tmp_test(""" import os -osx_cruft = %s -for key in osx_cruft: - try: del os.environ[key] - except: pass -print(os.environ["HERP"] + " " + str(len(os.environ))) -""" % osx_cruft) +for key in list(os.environ.keys()): + if key != "HERP": + del os.environ[key] +print(dict(os.environ)) +""") out = python(py.name, _env=env).strip() - self.assertEqual(out, "DERP 1") + self.assertEqual(out, "{'HERP': 'DERP'}") py = create_tmp_test(""" import os, sys sys.path.insert(0, os.getcwd()) import sh -osx_cruft = %s -for key in osx_cruft: - try: del os.environ[key] - except: pass -print(sh.HERP + " " + str(len(os.environ))) -""" % osx_cruft) +for key in list(os.environ.keys()): + if key != "HERP": + del os.environ[key] +print(dict(HERP=sh.HERP)) +""") out = python(py.name, _env=env, _cwd=THIS_DIR).strip() - self.assertEqual(out, "DERP 1") + self.assertEqual(out, "{'HERP': 'DERP'}") + # Test that _env also accepts os.environ which is a mpping but not a dict. + os.environ["HERP"] = "DERP" + out = python(py.name, _env=os.environ, _cwd=THIS_DIR).strip() + self.assertEqual(out, "{'HERP': 'DERP'}") def test_which(self): from sh import which, ls self.assertEqual(which("fjoawjefojawe"), None) self.assertEqual(which("ls"), str(ls)) - def test_which_paths(self): from sh import which py = create_tmp_test(""" @@ -563,6 +623,76 @@ found_path = which(test_name, [test_path]) self.assertEqual(found_path, py.name) + def test_no_close_fds(self): + # guarantee some extra fds in our parent process that don't close on exec. we have to explicitly do this + # because at some point (I believe python 3.4), python started being more stringent with closing fds to prevent + # security vulnerabilities. python 2.7, for example, doesn't set CLOEXEC on tempfile.TemporaryFile()s + # + # https://www.python.org/dev/peps/pep-0446/ + tmp = [tempfile.TemporaryFile() for i in range(10)] + for t in tmp: + flags = fcntl.fcntl(t.fileno(), fcntl.F_GETFD) + flags &= ~fcntl.FD_CLOEXEC + fcntl.fcntl(t.fileno(), fcntl.F_SETFD, flags) + + py = create_tmp_test(""" +import os +print(len(os.listdir("/dev/fd"))) +""") + out = python(py.name, _close_fds=False).strip() + # pick some number greater than 4, since it's hard to know exactly how many fds will be open/inherted in the + # child + self.assertGreater(int(out), 7) + + for t in tmp: + t.close() + + def test_close_fds(self): + # guarantee some extra fds in our parent process that don't close on exec. we have to explicitly do this + # because at some point (I believe python 3.4), python started being more stringent with closing fds to prevent + # security vulnerabilities. python 2.7, for example, doesn't set CLOEXEC on tempfile.TemporaryFile()s + # + # https://www.python.org/dev/peps/pep-0446/ + tmp = [tempfile.TemporaryFile() for i in range(10)] + for t in tmp: + flags = fcntl.fcntl(t.fileno(), fcntl.F_GETFD) + flags &= ~fcntl.FD_CLOEXEC + fcntl.fcntl(t.fileno(), fcntl.F_SETFD, flags) + + py = create_tmp_test(""" +import os +print(os.listdir("/dev/fd")) +""") + out = python(py.name).strip() + self.assertEqual(out, "['0', '1', '2', '3']") + + for t in tmp: + t.close() + + def test_pass_fds(self): + # guarantee some extra fds in our parent process that don't close on exec. we have to explicitly do this + # because at some point (I believe python 3.4), python started being more stringent with closing fds to prevent + # security vulnerabilities. python 2.7, for example, doesn't set CLOEXEC on tempfile.TemporaryFile()s + # + # https://www.python.org/dev/peps/pep-0446/ + tmp = [tempfile.TemporaryFile() for i in range(10)] + for t in tmp: + flags = fcntl.fcntl(t.fileno(), fcntl.F_GETFD) + flags &= ~fcntl.FD_CLOEXEC + fcntl.fcntl(t.fileno(), fcntl.F_SETFD, flags) + last_fd = tmp[-1].fileno() + + py = create_tmp_test(""" +import os +print(os.listdir("/dev/fd")) +""") + out = python(py.name, _pass_fds=[last_fd]).strip() + inherited = [0, 1, 2, 3, last_fd] + inherited_str = [str(i) for i in inherited] + self.assertEqual(out, str(inherited_str)) + + for t in tmp: + t.close() def test_no_arg(self): import pwd @@ -575,6 +705,16 @@ from sh import ls self.assertRaises(TypeError, ls, _iter=True, _piped=True) + def test_invalid_env(self): + from sh import ls + + exc = TypeError + if IS_PY2 and MINOR_VER == 6: + exc = ValueError + + self.assertRaises(exc, ls, _env="XXX") + self.assertRaises(exc, ls, _env={"foo": 123}) + self.assertRaises(exc, ls, _env={123: "bar"}) def test_exception(self): from sh import ErrorReturnCode_2 @@ -584,7 +724,6 @@ """) self.assertRaises(ErrorReturnCode_2, python, py.name) - def test_piped_exception1(self): from sh import ErrorReturnCode_2 @@ -595,7 +734,7 @@ exit(2) """) - py2 = create_tmp_test("") + py2 = create_tmp_test("") def fn(): list(python(python(py.name, _piped=True), "-u", py2.name, _iter=True)) @@ -612,30 +751,32 @@ exit(2) """) - py2 = create_tmp_test("") + py2 = create_tmp_test("") def fn(): python(python(py.name, _piped=True), "-u", py2.name) self.assertRaises(ErrorReturnCode_2, fn) - def test_command_not_found(self): from sh import CommandNotFound - def do_import(): from sh import aowjgoawjoeijaowjellll + def do_import(): + from sh import aowjgoawjoeijaowjellll # noqa: F401 + self.assertRaises(ImportError, do_import) def do_import(): import sh sh.awoefaowejfw + self.assertRaises(CommandNotFound, do_import) def do_import(): import sh sh.Command("ofajweofjawoe") - self.assertRaises(CommandNotFound, do_import) + self.assertRaises(CommandNotFound, do_import) def test_command_wrapper_equivalence(self): from sh import Command, ls, which @@ -663,11 +804,11 @@ from sh import gcc if IS_PY3: self.assertEqual(gcc._path, - gcc_file2.encode(sh.DEFAULT_ENCODING)) + gcc_file2.encode(sh.DEFAULT_ENCODING)) else: self.assertEqual(gcc._path, gcc_file2) self.assertEqual(gcc('no-error').stdout.strip(), - 'no-error'.encode("ascii")) + 'no-error'.encode("ascii")) finally: os.environ['PATH'] = save_path @@ -688,13 +829,12 @@ options, args = parser.parse_args() print(len(options.long_option.split())) """) - num_args = int(python(py.name, l="one two three")) + num_args = int(python(py.name, l="one two three")) # noqa: E741 self.assertEqual(num_args, 3) num_args = int(python(py.name, "-l", "one's two's three's")) self.assertEqual(num_args, 3) - def test_multiple_args_long_option(self): py = create_tmp_test(""" from optparse import OptionParser @@ -704,13 +844,12 @@ print(len(options.long_option.split())) """) num_args = int(python(py.name, long_option="one two three", - nothing=False)) + nothing=False)) self.assertEqual(num_args, 3) num_args = int(python(py.name, "--long-option", "one's two's three's")) self.assertEqual(num_args, 3) - def test_short_bool_option(self): py = create_tmp_test(""" from optparse import OptionParser @@ -723,7 +862,6 @@ self.assertTrue(python(py.name, s=False).strip() == "False") self.assertTrue(python(py.name).strip() == "False") - def test_long_bool_option(self): py = create_tmp_test(""" from optparse import OptionParser @@ -735,27 +873,34 @@ self.assertTrue(python(py.name, long_option=True).strip() == "True") self.assertTrue(python(py.name).strip() == "False") + def test_false_bool_ignore(self): + py = create_tmp_test(""" +import sys +print(sys.argv[1:]) +""") + test = True + self.assertEqual(python(py.name, test and "-n").strip(), "['-n']") + test = False + self.assertEqual(python(py.name, test and "-n").strip(), "[]") def test_composition(self): from sh import ls, wc - c1 = int(wc(ls("-A1"), l=True)) + c1 = int(wc(ls("-A1"), l=True)) # noqa: E741 c2 = len(os.listdir(".")) self.assertEqual(c1, c2) def test_incremental_composition(self): from sh import ls, wc - c1 = int(wc(ls("-A1", _piped=True), l=True).strip()) + c1 = int(wc(ls("-A1", _piped=True), l=True).strip()) # noqa: E741 c2 = len(os.listdir(".")) self.assertEqual(c1, c2) - def test_short_option(self): from sh import sh s1 = sh(c="echo test").strip() s2 = "test" self.assertEqual(s1, s2) - def test_long_option(self): py = create_tmp_test(""" from optparse import OptionParser @@ -783,7 +928,7 @@ print(options.long_option2.upper()) """) self.assertEqual(python(py.name, - {"long_option": "underscore"}).strip(), "UNDERSCORE") + {"long_option": "underscore"}).strip(), "UNDERSCORE") self.assertEqual(python(py.name, long_option="hyphen").strip(), "HYPHEN") @@ -805,7 +950,6 @@ out = python_baked().strip() self.assertEqual(out, correct) - def test_custom_separator_space(self): py = create_tmp_test(""" import sys @@ -816,7 +960,6 @@ out = python(py.name, opt, _long_sep=" ").strip() self.assertEqual(out, str(correct)) - def test_custom_long_prefix(self): py = create_tmp_test(""" import sys @@ -824,14 +967,21 @@ """) out = python(py.name, {"long-option": "underscore"}, - _long_prefix="-custom-").strip() + _long_prefix="-custom-").strip() self.assertEqual(out, "-custom-long-option=underscore") + out = python(py.name, {"long-option": True}, + _long_prefix="-custom-").strip() + self.assertEqual(out, "-custom-long-option") + # test baking too out = python.bake(py.name, {"long-option": "underscore"}, - _long_prefix="-baked-")().strip() + _long_prefix="-baked-")().strip() self.assertEqual(out, "-baked-long-option=underscore") + out = python.bake(py.name, {"long-option": True}, + _long_prefix="-baked-")().strip() + self.assertEqual(out, "-baked-long-option") def test_command_wrapper(self): from sh import Command, which @@ -839,12 +989,11 @@ ls = Command(which("ls")) wc = Command(which("wc")) - c1 = int(wc(ls("-A1"), l=True)) + c1 = int(wc(ls("-A1"), l=True)) # noqa: E741 c2 = len(os.listdir(".")) self.assertEqual(c1, c2) - def test_background(self): from sh import sleep import time @@ -854,21 +1003,20 @@ p = sleep(sleep_time, _bg=True) now = time.time() - self.assertTrue(now - start < sleep_time) + self.assertLess(now - start, sleep_time) p.wait() now = time.time() - self.assertTrue(now - start > sleep_time) - + self.assertGreater(now - start, sleep_time) def test_background_exception(self): from sh import ls, ErrorReturnCode_1, ErrorReturnCode_2 - p = ls("/ofawjeofj", _bg=True) # should not raise + p = ls("/ofawjeofj", _bg=True, _bg_exc=False) # should not raise exc_to_test = ErrorReturnCode_2 - if IS_OSX: exc_to_test = ErrorReturnCode_1 - self.assertRaises(exc_to_test, p.wait) # should raise - + if IS_MACOS: + exc_to_test = ErrorReturnCode_1 + self.assertRaises(exc_to_test, p.wait) # should raise def test_with_context(self): from sh import whoami @@ -886,9 +1034,8 @@ cmd1 = python.bake(py.name, _with=True) with cmd1: out = whoami() - self.assertTrue("with_context" in out) - self.assertTrue(getpass.getuser() in out) - + self.assertIn("with_context", out) + self.assertIn(getpass.getuser(), out) def test_with_context_args(self): from sh import whoami @@ -911,12 +1058,10 @@ out = whoami() self.assertTrue(getpass.getuser() == out.strip()) - with python(py.name, _with=True): out = whoami() self.assertTrue(out == "") - def test_binary_input(self): py = create_tmp_test(""" import sys @@ -927,7 +1072,6 @@ out = python(py.name, _in=data) self.assertEqual(out, "1234") - def test_err_to_out(self): py = create_tmp_test(""" import sys @@ -956,7 +1100,6 @@ self.assertEqual(stdout, "") self.assertEqual(os.read(master, 12), b"stdoutstderr") - def test_err_piped(self): py = create_tmp_test(""" import sys @@ -975,8 +1118,6 @@ out = python(python("-u", py.name, _piped="err"), "-u", py2.name) self.assertEqual(out, "stderr") - - def test_out_redirection(self): import tempfile @@ -991,28 +1132,25 @@ file_obj = tempfile.NamedTemporaryFile() out = python(py.name, _out=file_obj) - self.assertTrue(len(out) == 0) + self.assertEqual(len(out), 0) file_obj.seek(0) actual_out = file_obj.read() file_obj.close() - self.assertTrue(len(actual_out) != 0) - + self.assertNotEqual(len(actual_out), 0) # test with tee file_obj = tempfile.NamedTemporaryFile() out = python(py.name, _out=file_obj, _tee=True) - self.assertTrue(len(out) != 0) + self.assertGreater(len(out), 0) file_obj.seek(0) actual_out = file_obj.read() file_obj.close() - self.assertTrue(len(actual_out) != 0) - - + self.assertGreater(len(actual_out), 0) def test_err_redirection(self): import tempfile @@ -1031,9 +1169,9 @@ stderr = file_obj.read().decode() file_obj.close() - self.assertTrue(p.stdout == b"stdout") - self.assertTrue(stderr == "stderr") - self.assertTrue(len(p.stderr) == 0) + self.assertEqual(p.stdout, b"stdout") + self.assertEqual(stderr, "stderr") + self.assertEqual(len(p.stderr), 0) # now with tee file_obj = tempfile.NamedTemporaryFile() @@ -1043,10 +1181,9 @@ stderr = file_obj.read().decode() file_obj.close() - self.assertTrue(p.stdout == b"stdout") - self.assertTrue(stderr == "stderr") - self.assertTrue(len(p.stderr) != 0) - + self.assertEqual(p.stdout, b"stdout") + self.assertEqual(stderr, "stderr") + self.assertGreater(len(p.stderr), 0) def test_tty_tee(self): py = create_tmp_test(""" @@ -1071,26 +1208,24 @@ os.close(write) os.close(read) - def test_err_redirection_actual_file(self): - import tempfile - file_obj = tempfile.NamedTemporaryFile() - py = create_tmp_test(""" + import tempfile + file_obj = tempfile.NamedTemporaryFile() + py = create_tmp_test(""" import sys import os sys.stdout.write("stdout") sys.stderr.write("stderr") """) - stdout = python("-u", py.name, _err=file_obj.name).wait() - file_obj.seek(0) - stderr = file_obj.read().decode() - file_obj.close() - self.assertTrue(stdout == "stdout") - self.assertTrue(stderr == "stderr") + stdout = python("-u", py.name, _err=file_obj.name).wait() + file_obj.seek(0) + stderr = file_obj.read().decode() + file_obj.close() + self.assertTrue(stdout == "stdout") + self.assertTrue(stderr == "stderr") def test_subcommand_and_bake(self): - from sh import ls import getpass py = create_tmp_test(""" @@ -1104,9 +1239,8 @@ cmd1 = python.bake(py.name) out = cmd1.whoami() - self.assertTrue("subcommand" in out) - self.assertTrue(getpass.getuser() in out) - + self.assertIn("subcommand", out) + self.assertIn(getpass.getuser(), out) def test_multiple_bakes(self): py = create_tmp_test(""" @@ -1122,6 +1256,7 @@ import sys sys.stdout.write(str(sys.argv[1:])) """) + def arg_preprocess(args, kwargs): args.insert(0, "preprocessed") kwargs["a-kwarg"] = 123 @@ -1137,7 +1272,7 @@ ran = ls("-la").ran ft = ran.index("-h") - self.assertTrue("-la" in ran[ft:]) + self.assertIn("-la", ran[ft:]) def test_output_equivalence(self): from sh import whoami @@ -1147,7 +1282,6 @@ self.assertEqual(iam1, iam2) - # https://github.com/amoffat/sh/pull/252 def test_stdout_pipe(self): py = create_tmp_test(r""" @@ -1157,7 +1291,7 @@ """) read_fd, write_fd = os.pipe() - p = python(py.name, _out=write_fd, u=True) + python(py.name, _out=write_fd, u=True) def alarm(sig, action): self.fail("Timeout while reading from pipe") @@ -1171,7 +1305,6 @@ signal.alarm(0) signal.signal(signal.SIGALRM, signal.SIG_DFL) - def test_stdout_callback(self): py = create_tmp_test(""" import sys @@ -1180,15 +1313,14 @@ for i in range(5): print(i) """) stdout = [] + def agg(line): stdout.append(line) p = python("-u", py.name, _out=agg) p.wait() - self.assertTrue(len(stdout) == 5) - - + self.assertEqual(len(stdout), 5) def test_stdout_callback_no_wait(self): import time @@ -1204,17 +1336,16 @@ """) stdout = [] + def agg(line): stdout.append(line) - p = python("-u", py.name, _out=agg, _bg=True) + python("-u", py.name, _out=agg, _bg=True) # we give a little pause to make sure that the NamedTemporaryFile # exists when the python process actually starts time.sleep(.5) - self.assertTrue(len(stdout) != 5) - - + self.assertNotEqual(len(stdout), 5) def test_stdout_callback_line_buffered(self): py = create_tmp_test(""" @@ -1225,14 +1356,13 @@ """) stdout = [] + def agg(line): stdout.append(line) p = python("-u", py.name, _out=agg, _out_bufsize=1) p.wait() - self.assertTrue(len(stdout) == 5) - - + self.assertEqual(len(stdout), 5) def test_stdout_callback_line_unbuffered(self): py = create_tmp_test(""" @@ -1243,14 +1373,14 @@ """) stdout = [] + def agg(char): stdout.append(char) p = python("-u", py.name, _out=agg, _out_bufsize=0) p.wait() # + 5 newlines - self.assertTrue(len(stdout) == (len("herpderp") * 5 + 5)) - + self.assertEqual(len(stdout), len("herpderp") * 5 + 5) def test_stdout_callback_buffered(self): py = create_tmp_test(""" @@ -1261,14 +1391,13 @@ """) stdout = [] + def agg(chunk): stdout.append(chunk) p = python("-u", py.name, _out=agg, _out_bufsize=4) p.wait() - self.assertTrue(len(stdout) == (len("herp") / 2 * 5)) - - + self.assertEqual(len(stdout), len("herp") / 2 * 5) def test_stdout_callback_with_input(self): py = create_tmp_test(""" @@ -1283,14 +1412,13 @@ """) def agg(line, stdin): - if line.strip() == "4": stdin.put("derp\n") + if line.strip() == "4": + stdin.put("derp\n") p = python("-u", py.name, _out=agg, _tee=True) p.wait() - self.assertTrue("derp" in p) - - + self.assertIn("derp", p) def test_stdout_callback_exit(self): py = create_tmp_test(""" @@ -1301,18 +1429,18 @@ """) stdout = [] + def agg(line): line = line.strip() stdout.append(line) - if line == "2": return True + if line == "2": + return True p = python("-u", py.name, _out=agg, _tee=True) p.wait() - self.assertTrue("4" in p) - self.assertTrue("4" not in stdout) - - + self.assertIn("4", p) + self.assertNotIn("4", stdout) def test_stdout_callback_terminate(self): import signal @@ -1321,12 +1449,13 @@ import os import time -for i in range(5): +for i in range(5): print(i) time.sleep(.5) """) stdout = [] + def agg(line, stdin, process): line = line.strip() stdout.append(line) @@ -1344,10 +1473,8 @@ self.assertTrue(caught_signal) self.assertEqual(p.process.exit_code, -signal.SIGTERM) - self.assertTrue("4" not in p) - self.assertTrue("4" not in stdout) - - + self.assertNotIn("4", p) + self.assertNotIn("4", stdout) def test_stdout_callback_kill(self): import signal @@ -1357,12 +1484,13 @@ import os import time -for i in range(5): +for i in range(5): print(i) time.sleep(.5) """) stdout = [] + def agg(line, stdin, process): line = line.strip() stdout.append(line) @@ -1380,11 +1508,10 @@ self.assertTrue(caught_signal) self.assertEqual(p.process.exit_code, -signal.SIGKILL) - self.assertTrue("4" not in p) - self.assertTrue("4" not in stdout) + self.assertNotIn("4", p) + self.assertNotIn("4", stdout) def test_general_signal(self): - import signal from signal import SIGINT py = create_tmp_test(""" @@ -1396,7 +1523,7 @@ def sig_handler(sig, frame): print(10) exit(0) - + signal.signal(signal.SIGINT, sig_handler) for i in range(5): @@ -1406,6 +1533,7 @@ """) stdout = [] + def agg(line, stdin, process): line = line.strip() stdout.append(line) @@ -1419,14 +1547,13 @@ self.assertEqual(p.process.exit_code, 0) self.assertEqual(p, "0\n1\n2\n3\n10\n") - def test_iter_generator(self): py = create_tmp_test(""" import sys import os import time -for i in range(42): +for i in range(42): print(i) sys.stdout.flush() """) @@ -1434,15 +1561,16 @@ out = [] for line in python(py.name, _iter=True): out.append(int(line.strip())) - self.assertTrue(len(out) == 42 and sum(out) == 861) + self.assertEqual(len(out), 42) + self.assertEqual(sum(out), 861) def test_iter_unicode(self): # issue https://github.com/amoffat/sh/issues/224 - test_string = "\xe4\xbd\x95\xe4\xbd\x95\n" * 150 # len > buffer_s + test_string = "\xe4\xbd\x95\xe4\xbd\x95\n" * 150 # len > buffer_s txt = create_tmp_test(test_string) for line in sh.cat(txt.name, _iter=True): break - self.assertTrue(len(line) < 1024) + self.assertLess(len(line), 1024) def test_nonblocking_iter(self): from errno import EWOULDBLOCK @@ -1460,7 +1588,7 @@ count += 1 else: value = line - self.assertTrue(count > 0) + self.assertGreater(count, 0) self.assertEqual(value, "stdout") py = create_tmp_test(""" @@ -1477,36 +1605,30 @@ count += 1 else: value = line - self.assertTrue(count > 0) + self.assertGreater(count, 0) self.assertEqual(value, "stderr") - - - def test_for_generator_to_err(self): py = create_tmp_test(""" import sys import os -for i in range(42): +for i in range(42): sys.stderr.write(str(i)+"\\n") """) out = [] for line in python("-u", py.name, _iter="err"): out.append(line) - self.assertTrue(len(out) == 42) + self.assertEqual(len(out), 42) # verify that nothing is going to stdout out = [] for line in python("-u", py.name, _iter="out"): out.append(line) - self.assertTrue(len(out) == 0) - + self.assertEqual(len(out), 0) def test_sigpipe(self): - import sh - py1 = create_tmp_test(""" import sys import os @@ -1544,10 +1666,7 @@ self.assertEqual(-p1.exit_code, signal.SIGPIPE) self.assertEqual(p2.exit_code, 0) - def test_piped_generator(self): - from sh import tr - from string import ascii_uppercase import time py1 = create_tmp_test(""" @@ -1572,44 +1691,43 @@ print(line.strip().upper()) """) - times = [] last_received = None letters = "" for line in python(python("-u", py1.name, _piped="out"), "-u", - py2.name, _iter=True): - if not letters: - start = time.time() + py2.name, _iter=True): letters += line.strip() now = time.time() - if last_received: times.append(now - last_received) + if last_received: + times.append(now - last_received) last_received = now self.assertEqual("ANDREW", letters) self.assertTrue(all([t > .3 for t in times])) - def test_generator_and_callback(self): py = create_tmp_test(""" import sys import os for i in range(42): - sys.stderr.write(str(i * 2)+"\\n") + sys.stderr.write(str(i * 2)+"\\n") print(i) """) stderr = [] - def agg(line): stderr.append(int(line.strip())) - out = [] - for line in python("-u", py.name, _iter=True, _err=agg): out.append(line) + def agg(line): + stderr.append(int(line.strip())) - self.assertTrue(len(out) == 42) - self.assertTrue(sum(stderr) == 1722) + out = [] + for line in python("-u", py.name, _iter=True, _err=agg): + out.append(line) + self.assertEqual(len(out), 42) + self.assertEqual(sum(stderr), 1722) def test_cast_bg(self): py = create_tmp_test(""" @@ -1634,10 +1752,23 @@ def test_fg(self): py = create_tmp_test("exit(0)") - # notice we're using `sh.python`, and not `python`. this is because + # notice we're using `system_python`, and not `python`. this is because # `python` has an env baked into it, and we want `_env` to be None for # coverage - sh.python(py.name, _fg=True) + system_python(py.name, _fg=True) + + def test_fg_false(self): + """ https://github.com/amoffat/sh/issues/520 """ + py = create_tmp_test("print('hello')") + buf = StringIO() + python(py.name, _fg=False, _out=buf) + self.assertEqual(buf.getvalue(), "hello\n") + + def test_fg_true(self): + """ https://github.com/amoffat/sh/issues/520 """ + py = create_tmp_test("print('hello')") + buf = StringIO() + self.assertRaises(TypeError, python, py.name, _fg=True, _out=buf) def test_fg_env(self): py = create_tmp_test(""" @@ -1649,7 +1780,7 @@ env = os.environ.copy() env["EXIT"] = "3" self.assertRaises(sh.ErrorReturnCode_3, python, py.name, _fg=True, - _env=env) + _env=env) def test_fg_alternative(self): py = create_tmp_test("exit(0)") @@ -1681,6 +1812,24 @@ self.assertEqual(str(pwd(_cwd="/tmp")), realpath("/tmp") + "\n") self.assertEqual(str(pwd(_cwd="/etc")), realpath("/etc") + "\n") + def test_cwd_fg(self): + td = realpath(tempfile.mkdtemp()) + py = create_tmp_test(""" +import sh +import os +from os.path import realpath +orig = realpath(os.getcwd()) +print(orig) +sh.pwd(_cwd="{newdir}", _fg=True) +print(realpath(os.getcwd())) +""".format(newdir=td)) + + orig, newdir, restored = python(py.name).strip().split("\n") + newdir = realpath(newdir) + self.assertEqual(newdir, td) + self.assertEqual(orig, restored) + self.assertNotEqual(orig, newdir) + os.rmdir(td) def test_huge_piped_data(self): from sh import tr @@ -1695,7 +1844,6 @@ out = tr(tr("[:lower:]", "[:upper:]", _in=data), "[:upper:]", "[:lower:]") self.assertTrue(out == data) - def test_tty_input(self): py = create_tmp_test(""" import sys @@ -1718,7 +1866,8 @@ def password_enterer(line, stdin): line = line.strip() - if not line: return + if not line: + return if line == "password?": stdin.put(test_pw + "\n") @@ -1734,7 +1883,6 @@ response = python(py.name) self.assertEqual(response, "no tty attached!\n") - def test_tty_output(self): py = create_tmp_test(""" import sys @@ -1754,7 +1902,6 @@ out = python(py.name, _tty_out=False) self.assertEqual(out, "no tty attached") - def test_stringio_output(self): from sh import echo @@ -1774,7 +1921,6 @@ echo("-n", "testing 123", _out=out) self.assertEqual(out.getvalue().decode(), "testing 123") - def test_stringio_input(self): from sh import cat @@ -1785,17 +1931,15 @@ out = cat(_in=input) self.assertEqual(out, "herpderp") - def test_internal_bufsize(self): from sh import cat - output = cat(_in="a"*1000, _internal_bufsize=100, _out_bufsize=0) + output = cat(_in="a" * 1000, _internal_bufsize=100, _out_bufsize=0) self.assertEqual(len(output), 100) - output = cat(_in="a"*1000, _internal_bufsize=50, _out_bufsize=2) + output = cat(_in="a" * 1000, _internal_bufsize=50, _out_bufsize=2) self.assertEqual(len(output), 100) - def test_change_stdout_buffering(self): py = create_tmp_test(""" import sys @@ -1822,9 +1966,11 @@ "newline_buffer_success": False, "unbuffered_success": False, } + def interact(line, stdin, process): line = line.strip() - if not line: return + if not line: + return if line == "switch buffering": d["newline_buffer_success"] = True @@ -1843,7 +1989,6 @@ self.assertTrue(d["newline_buffer_success"]) self.assertTrue(d["unbuffered_success"]) - def test_callable_interact(self): py = create_tmp_test(""" import sys @@ -1853,6 +1998,7 @@ class Callable(object): def __init__(self): self.line = None + def __call__(self, line): self.line = line @@ -1860,12 +2006,8 @@ python(py.name, _out=cb) self.assertEqual(cb.line, "line1") - def test_encoding(self): - return - raise NotImplementedError("what's the best way to test a different \ -'_encoding' special keyword argument?") - + return self.skipTest("what's the best way to test a different '_encoding' special keyword argument?") def test_timeout(self): import sh @@ -1876,20 +2018,30 @@ started = time() try: sh.sleep(sleep_for, _timeout=timeout).wait() - except sh.TimeoutException: - pass + except sh.TimeoutException as e: + assert 'sleep 3' in e.full_cmd else: self.fail("no timeout exception") elapsed = time() - started - self.assertTrue(abs(elapsed - timeout) < 0.5) - + self.assertLess(abs(elapsed - timeout), 0.5) def test_timeout_overstep(self): started = time.time() sh.sleep(1, _timeout=5) elapsed = time.time() - started - self.assertTrue(abs(elapsed - 1) < 0.5) + self.assertLess(abs(elapsed - 1), 0.5) + def test_timeout_wait(self): + p = sh.sleep(3, _bg=True) + self.assertRaises(sh.TimeoutException, p.wait, timeout=1) + + def test_timeout_wait_overstep(self): + p = sh.sleep(1, _bg=True) + p.wait(timeout=5) + + def test_timeout_wait_negative(self): + p = sh.sleep(3, _bg=True) + self.assertRaises(RuntimeError, p.wait, timeout=-3) def test_binary_pipe(self): binary = b'\xec;\xedr\xdbF' @@ -1913,7 +2065,6 @@ out = python(python(py1.name), py2.name) self.assertEqual(out.stdout, binary) - # designed to trigger the "... (%d more, please see e.stdout)" output # of the ErrorReturnCode class def test_failure_with_large_output(self): @@ -1940,7 +2091,6 @@ self.assertRaises(ErrorReturnCode, ls, test) - def test_no_out(self): py = create_tmp_test(""" import sys @@ -1953,6 +2103,7 @@ self.assertTrue(p.process._pipe_queue.empty()) def callback(line): pass + p = python(py.name, _out=callback) self.assertEqual(p.stdout, b"") self.assertEqual(p.stderr, b"stderr") @@ -1963,7 +2114,6 @@ self.assertEqual(p.stderr, b"stderr") self.assertFalse(p.process._pipe_queue.empty()) - def test_tty_stdin(self): py = create_tmp_test(""" import sys @@ -1973,7 +2123,6 @@ out = python(py.name, _in="test\n", _tty_in=True) self.assertEqual("test\n", out) - def test_no_err(self): py = create_tmp_test(""" import sys @@ -1986,6 +2135,7 @@ self.assertFalse(p.process._pipe_queue.empty()) def callback(line): pass + p = python(py.name, _err=callback) self.assertEqual(p.stderr, b"") self.assertEqual(p.stdout, b"stdout") @@ -1996,7 +2146,6 @@ self.assertEqual(p.stdout, b"stdout") self.assertFalse(p.process._pipe_queue.empty()) - def test_no_pipe(self): from sh import ls @@ -2006,6 +2155,7 @@ # calling a command with a callback should not def callback(line): pass + p = ls(_out=callback) self.assertTrue(p.process._pipe_queue.empty()) @@ -2013,7 +2163,6 @@ p = ls(_no_pipe=True) self.assertTrue(p.process._pipe_queue.empty()) - def test_decode_error_handling(self): from functools import partial @@ -2029,13 +2178,14 @@ sys.stdout.write("te漢字st") """) fn = partial(python, py.name, _encoding="ascii") + def s(fn): str(fn()) + self.assertRaises(UnicodeDecodeError, s, fn) p = python(py.name, _encoding="ascii", _decode_errors="ignore") self.assertEqual(p, "test") - def test_signal_exception(self): from sh import SignalException_15 @@ -2050,7 +2200,6 @@ self.assertRaises(SignalException_15, throw_terminate_signal) - def test_signal_group(self): child = create_tmp_test(""" import time @@ -2058,8 +2207,10 @@ """) parent = create_tmp_test(""" +import sys import sh -p = sh.python("{child_file}", _bg=True, _new_session=False) +python = sh.Command(sys.executable) +p = python("{child_file}", _bg=True, _new_session=False) print(p.pid) print(p.process.pgid) p.wait() @@ -2080,7 +2231,6 @@ def assert_dead(pid): self.assert_oserror(errno.ESRCH, os.kill, pid, 0) - # first let's prove that calling regular SIGKILL on the parent does # nothing to the child, since the child was launched in the same process # group (_new_session=False) and the parent is not a controlling process @@ -2097,7 +2247,6 @@ self.assertRaises(sh.SignalException_SIGKILL, p.wait) assert_dead(child_pid) - # now let's prove that killing the process group kills both the parent # and the child p, child_pid, child_pgid, parent_pid, parent_pgid = launch() @@ -2110,7 +2259,6 @@ assert_dead(parent_pid) assert_dead(child_pid) - def test_pushd(self): """ test basic pushd functionality """ old_wd1 = sh.pwd().strip() @@ -2131,12 +2279,9 @@ self.assertEqual(new_wd1, tempdir) self.assertEqual(new_wd2, tempdir) - - def test_pushd_cd(self): """ test that pushd works like pushd/popd with built-in cd correctly """ import sh - from sh import mkdir child = realpath(tempfile.mkdtemp()) try: @@ -2152,7 +2297,7 @@ def test_cd_homedir(self): orig = os.getcwd() - my_dir = os.path.expanduser("~") + my_dir = os.path.realpath(os.path.expanduser("~")) # Use realpath because homedir may be a symlink sh.cd() self.assertNotEqual(orig, os.getcwd()) @@ -2162,11 +2307,9 @@ from sh import ls # sanity check - non_exist_dir = join(tempdir, "aowjgoahewro") + non_exist_dir = join(tempdir, "aowjgoahewro") self.assertFalse(exists(non_exist_dir)) - - self.assertRaises(OSError, ls, _cwd=non_exist_dir) - + self.assertRaises(sh.ForkException, ls, _cwd=non_exist_dir) # https://github.com/amoffat/sh/issues/176 def test_baked_command_can_be_printed(self): @@ -2175,7 +2318,6 @@ ll = ls.bake("-l") self.assertTrue(str(ll).endswith("/ls -l")) - # https://github.com/amoffat/sh/issues/185 def test_done_callback(self): import time @@ -2207,21 +2349,20 @@ wait_elapsed = time.time() - wait_start self.assertTrue(callback.called) - self.assertTrue(abs(wait_elapsed - 1.0) < 1.0) + self.assertLess(abs(wait_elapsed - 1.0), 1.0) self.assertEqual(callback.exit_code, 0) self.assertTrue(callback.success) - def test_fork_exc(self): from sh import ForkException py = create_tmp_test("") + def fail(): raise RuntimeError("nooo") self.assertRaises(ForkException, python, py.name, _preexec_fn=fail) - def test_new_session(self): from threading import Event @@ -2237,7 +2378,6 @@ time.sleep(0.5) """) - event = Event() def handle(line, stdin, p): @@ -2262,10 +2402,8 @@ p.wait() self.assertTrue(event.is_set()) - event.clear() - def handle(line, stdin, p): pid, pgid, sid = line.strip().split(",") pid = int(pid) @@ -2290,8 +2428,6 @@ p.wait() self.assertTrue(event.is_set()) - - def test_done_cb_exc(self): from sh import ErrorReturnCode @@ -2299,6 +2435,7 @@ def __init__(self): self.called = False self.success = None + def __call__(self, p, success, exit_code): self.success = success self.called = True @@ -2337,7 +2474,6 @@ self.assertEqual("0123", out) def test_stdin_unbuffered_bufsize(self): - import sh from time import sleep # this tries to receive some known data and measures the time it takes @@ -2369,19 +2505,16 @@ sleep(1) yield "done" - out = python(py.name, _in=create_stdin(), _in_bufsize=0) word1, time1, word2, time2, _ = out.split("\n") time1 = float(time1) time2 = float(time2) self.assertEqual(word1, "testing") - self.assertTrue(abs(1-time1) < 0.5) + self.assertLess(abs(1 - time1), 0.5) self.assertEqual(word2, "done") - self.assertTrue(abs(1-time2) < 0.5) - + self.assertLess(abs(1 - time2), 0.5) def test_stdin_newline_bufsize(self): - import sh from time import sleep # this tries to receive some known data and measures the time it takes @@ -2416,16 +2549,14 @@ sleep(1) yield "done\n" - out = python(py.name, _in=create_stdin(), _in_bufsize=1) word1, time1, word2, time2, _ = out.split("\n") time1 = float(time1) time2 = float(time2) self.assertEqual(word1, "testing") - self.assertTrue(abs(1-time1) < 0.5) + self.assertLess(abs(1 - time1), 0.5) self.assertEqual(word2, "done") - self.assertTrue(abs(1-time2) < 0.5) - + self.assertLess(abs(1 - time2), 0.5) def test_custom_timeout_signal(self): from sh import TimeoutException @@ -2442,7 +2573,6 @@ else: self.fail("we should have handled a TimeoutException") - def test_append_stdout(self): py = create_tmp_test(""" import sys @@ -2464,7 +2594,6 @@ out = python.bake(py.name).bake_() self.assertEqual("bake", out) - def test_no_proc_no_attr(self): py = create_tmp_test("") with python(py.name) as p: @@ -2479,20 +2608,21 @@ """) output = [] + def fn(foo, line): output.append((foo, int(line.strip()))) log_line = partial(fn, "hello") - out = python(py.name, _out=log_line) + python(py.name, _out=log_line) self.assertEqual(output, [("hello", i) for i in range(10)]) - output = [] + def fn(foo, line, stdin, proc): output.append((foo, int(line.strip()))) log_line = partial(fn, "hello") - out = python(py.name, _out=log_line) + python(py.name, _out=log_line) self.assertEqual(output, [("hello", i) for i in range(10)]) # https://github.com/amoffat/sh/issues/266 @@ -2525,7 +2655,8 @@ child_file = sys.argv[1] output_file = sys.argv[2] -os.spawnlp(os.P_NOWAIT, "python", "python", child_file, output_file) +python_name = os.path.basename(sys.executable) +os.spawnlp(os.P_NOWAIT, python_name, python_name, child_file, output_file) time.sleep(1) # give child a chance to set up """) @@ -2536,7 +2667,6 @@ out = output_file.readlines()[0] self.assertEqual(out, b"made it!\n") - def test_unchecked_producer_failure(self): from sh import ErrorReturnCode_2 @@ -2556,7 +2686,6 @@ direct_pipe = python(producer.name, _piped=True) self.assertRaises(ErrorReturnCode_2, python, direct_pipe, consumer.name) - def test_unchecked_pipeline_failure(self): # similar to test_unchecked_producer_failure, but this # tests a multi-stage pipeline @@ -2603,7 +2732,6 @@ self.assertEqual(test(), "some output") self.assertRaises(sh.CommandNotFound, fn) - def test_patch_command(self): def fn(): return sh.afowejfow() @@ -2618,6 +2746,26 @@ class MiscTests(BaseTests): + def test_pickling(self): + import pickle + + py = create_tmp_test(""" +import sys +sys.stdout.write("some output") +sys.stderr.write("some error") +exit(1) +""") + + try: + python(py.name) + except sh.ErrorReturnCode as e: + restored = pickle.loads(pickle.dumps(e)) + self.assertEqual(restored.stdout, b"some output") + self.assertEqual(restored.stderr, b"some error") + self.assertEqual(restored.exit_code, 1) + else: + self.fail("Didn't get an exception") + @requires_poller("poll") def test_fd_over_1024(self): py = create_tmp_test("""print("hi world")""") @@ -2636,38 +2784,29 @@ os.close(master) os.close(slave) - def test_args_deprecated(self): self.assertRaises(DeprecationWarning, sh.args, _env={}) - def test_cant_import_all(self): - def go(): - # we have to use exec, because in py3, this syntax raises a - # SyntaxError upon compilation - exec("from sh import *") - self.assertRaises(RuntimeError, go) - def test_percent_doesnt_fail_logging(self): """ test that a command name doesn't interfere with string formatting in the internal loggers """ py = create_tmp_test(""" print("cool") """) - out = python(py.name, "%") - out = python(py.name, "%%") - out = python(py.name, "%%%") - + python(py.name, "%") + python(py.name, "%%") + python(py.name, "%%%") # TODO # for some reason, i can't get a good stable baseline measured in this test # on osx. so skip it for now if osx - @not_osx + @not_macos @requires_progs("lsof") def test_no_fd_leak(self): import sh import os from itertools import product - + # options whose combinations can possibly cause fd leaks kwargs = { "_tty_out": (True, False), @@ -2691,10 +2830,11 @@ opt_dict[key] = val yield opt_dict - test_pid = os.getpid() + def get_num_fds(): lines = sh.lsof(p=test_pid).strip().split("\n") + def test(line): line = line.upper() return "CHR" in line or "PIPE" in line @@ -2703,6 +2843,7 @@ return len(lines) - 1 py = create_tmp_test("") + def test_command(**opts): python(py.name, **opts) @@ -2714,14 +2855,12 @@ num_fds = get_num_fds() self.assertEqual(baseline, num_fds) - for opts in get_opts(kwargs): for i in xrange(2): test_command(**opts) num_fds = get_num_fds() self.assertEqual(baseline, num_fds, (baseline, num_fds, opts)) - def test_pushd_thread_safety(self): import threading import time @@ -2756,7 +2895,6 @@ os.rmdir(temp1) os.rmdir(temp2) - def test_stdin_nohang(self): py = create_tmp_test(""" print("hi") @@ -2769,10 +2907,11 @@ def test_unicode_path(self): from sh import Command - py = create_tmp_test("""#!/usr/bin/env python + python_name = os.path.basename(sys.executable) + py = create_tmp_test("""#!/usr/bin/env {0} # -*- coding: utf8 -*- print("字") -""", "字", delete=False) +""".format(python_name), prefix="字", delete=False) try: py.close() @@ -2796,7 +2935,6 @@ finally: os.unlink(py.name) - # https://github.com/amoffat/sh/issues/121 def test_wraps(self): from sh import ls @@ -2814,7 +2952,6 @@ self.assertEqual(sig, SignalException_SIGQUIT) - def test_change_log_message(self): py = create_tmp_test(""" print("cool") @@ -2838,13 +2975,10 @@ self.assertTrue(loglines, "Log handler captured no messages?") self.assertTrue(loglines[0].startswith("Hi! I ran something")) - # https://github.com/amoffat/sh/issues/273 def test_stop_iteration_doesnt_block(self): """ proves that calling calling next() on a stopped iterator doesn't hang. """ - - import sh py = create_tmp_test(""" print("cool") """) @@ -2857,7 +2991,6 @@ # https://github.com/amoffat/sh/issues/195 def test_threaded_with_contexts(self): - import sh import threading import time @@ -2875,11 +3008,11 @@ def f1(): with p1: time.sleep(1) - results[0] = str(sh.python("one")) + results[0] = str(system_python("one")) def f2(): with p2: - results[1] = str(sh.python("two")) + results[1] = str(system_python("two")) t1 = threading.Thread(target=f1) t1.start() @@ -2896,12 +3029,12 @@ ] self.assertEqual(results, correct) - # https://github.com/amoffat/sh/pull/292 def test_eintr(self): import signal def handler(num, frame): pass + signal.signal(signal.SIGALRM, handler) py = create_tmp_test(""" @@ -2913,10 +3046,9 @@ p.wait() - class StreamBuffererTests(unittest.TestCase): def test_unbuffered(self): - from sh import _disable_whitelist, StreamBufferer + from sh import StreamBufferer b = StreamBufferer(0) self.assertEqual(b.process(b"test"), [b"test"]) @@ -2925,7 +3057,7 @@ self.assertEqual(b.flush(), b"") def test_newline_buffered(self): - from sh import _disable_whitelist, StreamBufferer + from sh import StreamBufferer b = StreamBufferer(1) self.assertEqual(b.process(b"testing\none\ntwo"), [b"testing\n", b"one\n"]) @@ -2933,7 +3065,7 @@ self.assertEqual(b.flush(), b"four") def test_chunk_buffered(self): - from sh import _disable_whitelist, StreamBufferer + from sh import StreamBufferer b = StreamBufferer(10) self.assertEqual(b.process(b"testing\none\ntwo"), [b"testing\non"]) @@ -2953,7 +3085,7 @@ def test_no_interfere1(self): import sh out = StringIO() - _sh = sh(_out=out) + _sh = sh(_out=out) # noqa: F841 from _sh import echo echo("-n", "TEST") self.assertEqual("TEST", out.getvalue()) @@ -2969,7 +3101,7 @@ import sh out = StringIO() from sh import echo - _sh = sh(_out=out) + _sh = sh(_out=out) # noqa: F841 echo("-n", "TEST") self.assertEqual("", out.getvalue()) @@ -2986,11 +3118,14 @@ import sh out = StringIO() _sh = sh(_out=out) + def nested1(): _sh.echo("-n", "TEST1") + def nested2(): import sh sh.echo("-n", "TEST2") + nested1() nested2() self.assertEqual("TEST1", out.getvalue()) @@ -3006,15 +3141,41 @@ def test_importer_detects_module_name(self): import sh _sh = sh() - omg = _sh - from omg import python + omg = _sh # noqa: F841 + from omg import cat # noqa: F401 def test_importer_only_works_with_sh(self): def unallowed_import(): - _os = os - from _os import path + _os = os # noqa: F841 + from _os import path # noqa: F401 + self.assertRaises(ImportError, unallowed_import) + def test_reimport_from_cli(self): + # The REPL and CLI both need special handling to create an execution context that is safe to + # reimport + if IS_PY3: + cmdstr = '; '.join(('import sh, io, sys', + 'out = io.StringIO()', + '_sh = sh(_out=out)', + 'import _sh', + '_sh.echo("-n", "TEST")', + 'sys.stderr.write(out.getvalue())', + )) + else: + cmdstr = '; '.join(('import sh, StringIO, sys', + 'out = StringIO.StringIO()', + '_sh = sh(_out=out)', + 'import _sh', + '_sh.echo("-n", "TEST")', + 'sys.stderr.write(out.getvalue())', + )) + + err = StringIO() + + python('-c', cmdstr, _err=err) + self.assertEqual('TEST', err.getvalue()) + if __name__ == "__main__": root = logging.getLogger() @@ -3027,7 +3188,6 @@ test_kwargs["failfast"] = True test_kwargs["verbosity"] = 2 - try: # if we're running a specific test, we can let unittest framework figure out # that test and run it itself. it will also handle setting the return code diff -Nru python-sh-1.12.14/tox.ini python-sh-1.14.1/tox.ini --- python-sh-1.12.14/tox.ini 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/tox.ini 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,13 @@ +[tox] +# virtualenv for py26 is broken, so don't put it here +envlist = py{27,31,32,33,34,35,36,37,38},docs + +[testenv] +deps = -r requirements-dev.txt +commands = + python sh.py tox + +[testenv:docs] +basepython = python3 +commands = + python setup.py check --restructuredtext --metadata --strict diff -Nru python-sh-1.12.14/.travis.yml python-sh-1.14.1/.travis.yml --- python-sh-1.12.14/.travis.yml 1970-01-01 00:00:00.000000000 +0000 +++ python-sh-1.14.1/.travis.yml 2020-10-24 17:34:57.000000000 +0000 @@ -0,0 +1,27 @@ +os: + - linux + +language: python + +python: + # - 2.6 No longer supported on Travis + - 2.7 + # - 3.3 No longer supported on Travis + - 3.4 + - 3.5 + - 3.6 + - 3.7 + - 3.8 + - pypy + - pypy3.5 + +before_script: + - pip install -r requirements-dev.txt + +script: + - python sh.py travis + - python -m flake8 sh.py test.py + - if python -c 'import sys; sys.exit(int(not sys.version_info >= (3, 5)))' ; then python setup.py check --restructuredtext --metadata --strict ; fi + +after_success: + - coveralls