The sarge package provides a wrapper for subprocess which provides command pipeline functionality.

Hi

below code always return [0, 0, None] and command never finish : seems have problem with sleep ...

cmd = 'echo "Hello " && sleep 2  && echo "Finish " '
p = run(cmd, async_=True)

Original report by Anonymous.

When using redirection that leads to an ioerror, sarge can hang waiting for spawned processes that will never complete due to waiting on a full pipe that no one is reading.

A simple test case:

#!python

import sarge
pipeline = sarge.run('head -c 65K /dev/zero | cat > /does/not/exist')

The first command hangs on a write to the pipe, which is never closed and sarge hangs on a wait() for this process that will never finish.

Original report by Dmitry Malinovsky (Bitbucket: malinoff, GitHub: malinoff).

In [1]: from sarge import run

In [2]: p = run('exit 1', shell=True)

In [3]: p.returncode
Out[3]: 0

Original report by Allan Lei (Bitbucket: allanlei, GitHub: allanlei).

I get this error for any commands I run.

• Windows 7 Pro 64
• Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32
• pip install sarge==0.1.2

Simple run

#!python

sarge.run('echo abc')

Trying the test_sarge.py script from repo

#!bash

PS C:\Users\allan.lei\Desktop> python test_sarge.py
E.EEEEE.E...EEE..EEEEEEEE....EException in thread Thread-3:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1055, in run_node
    result = getattr(self, method)(node, input, async)
  File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1201, in run_command_node
    node.cmd.run(input=input, async=async)
  File "C:\Python27\lib\site-packages\sarge\__init__.py", line 593, in run
    self.process = p = Popen(self.args, **self.kwargs)
  File "C:\Python27\lib\subprocess.py", line 701, in __init__
    errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
  File "C:\Python27\lib\site-packages\sarge\__init__.py", line 481, in _get_handles
    stderr)
ValueError: need more than 2 values to unpack

Exception in thread Thread-4:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1055, in run_node
    result = getattr(self, method)(node, input, async)
  File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1289, in run_list_node
    self.run_node(curr, input, async=use_async)
  File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1055, in run_node
    result = getattr(self, method)(node, input, async)
  File "C:\Python27\lib\site-packages\sarge\__init__.py", line 1201, in run_command_node
    node.cmd.run(input=input, async=async)
  File "C:\Python27\lib\site-packages\sarge\__init__.py", line 593, in run
    self.process = p = Popen(self.args, **self.kwargs)
  File "C:\Python27\lib\subprocess.py", line 701, in __init__
    errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
  File "C:\Python27\lib\site-packages\sarge\__init__.py", line 481, in _get_handles
    stderr)
ValueError: need more than 2 values to unpack

Original report by David Lowe (Bitbucket: [David D Lowe](https://bitbucket.org/David D Lowe), ).

Current behaviour:

>>> shell_quote("'ab?'")
 "'ab?'"
>>> print(shell_quote("'ab?'"))
'ab?'

Expected behaviour:

 >>> shell_quote("'ab?'")
 '"\'ab?\'"'
 >>> print(shell_quote("'ab?'"))
 "'ab?'"

For example, to create a file named 'ab?', that is, a filename that consists of a single quote character, the a character, the b character, the ? character and the single quote character, in bash you would type the following:

 $ touch "'ab?'"

And not the following:

 $ touch 'ab?'

Original report by Jeet Ray (Bitbucket: shadowrylander, GitHub: shadowrylander).

Hello! Would it be possible to use the feeder with capture_both? I would like to store the result in a variable until I can display or return it manually! My current code is something like this:

from sarge import Feeder, get_stdout, capture_both

feeder = Feeder()
for line in capture_both("cowsay", input=feeder, async_=True).stdout:
    print(line.decode("utf-8").rstrip())
feeder.feed(get_stdout("fortune"))
feeder.close()

Thank you kindly for the help!

Original report by CDuv (Bitbucket: CDuv, GitHub: CDuv).

In want to use the sarge library to execute a simple cmd1 | cmd2 shell command from my Python script an grab it's output (both stdout and stderr) live while the shell command executes. I was previously using subprocess.Popen() + subprocess.PIPE + communicate() but the output was buffered.

As a working base I tested the progress.py code detailed in the tutorial.

File "test_sarge_live.py":

import optparse # because of 2.6 support
import sys
import threading
import time
import logging

from sarge import capture_stdout

logging.basicConfig(level=logging.DEBUG, filename='/tmp/test_sarge_live.log',
                    filemode='w', format='%(asctime)s %(threadName)-10s %(name)-15s %(lineno)4d %(message)s')

def progress(capture, options):
    lines_seen = 0
    messages = {
        'line 25\n': 'Getting going ...\n',
        'line 50\n': 'Well on the way ...\n',
        'line 75\n': 'Almost there ...\n',
    }
    while True:
        s = capture.readline()
        if not s and lines_seen:
            break
        if options.dots:
            sys.stderr.write('.')
        else:
            msg = messages.get(s)
            if msg:
                sys.stderr.write(msg)
        lines_seen += 1
    if options.dots:
        sys.stderr.write('\n')
    sys.stderr.write('Done - %d lines seen.\n' % lines_seen)

def main():
    parser = optparse.OptionParser()
    parser.add_option('-n', '--no-dots', dest='dots', default=True,
                      action='store_false', help='Show dots for progress')
    options, args = parser.parse_args()

    #~ p = capture_stdout('ncat -k -l -p 42421', async=True)
    p = capture_stdout('python lister.py -d 0.1 -c 100', async=True)

    t = threading.Thread(target=progress, args=(p.stdout, options))
    t.start()

    while(p.returncodes[0] is None):
        # We could do other useful work here. If we have no useful
        # work to do here, we can call readline() and process it
        # directly in this loop, instead of creating a thread to do it in.
        p.commands[0].poll()
        time.sleep(0.05)
    t.join()

if __name__ == '__main__':
    sys.exit(main())

But running it gives very inconsistent output:

On Ubuntu v16.04 using Python v2.7.12 and sarge v0.1.4 I get theses (at random):

• This ('NoneType' object has no attribute 'returncode' + sys.excepthook is missing)

Traceback (most recent call last):
File "test_sarge_live.py", line 55, in <module>
  sys.exit(main())
File "test_sarge_live.py", line 46, in main
  while(p.returncodes[0] is None):
File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1072, in returncodes
  result = [c.process.returncode for c in self.commands]
AttributeError: 'NoneType' object has no attribute 'returncode'
..
Done - 2 lines seen.
close failed in file object destructor:                                                                                                                                                                                                                         
sys.excepthook is missing
lost sys.stderr

• Or also this (no 'NoneType' object has no attribute 'returncode')

close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

• Or even this:

Traceback (most recent call last):
.  File "test_sarge_live.py", line 55, in <module>
  
sys.exit(main())Done - 1 lines seen.

File "test_sarge_live.py", line 46, in main
  while(p.returncodes[0] is None):
IndexError: list index out of range
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
File "/usr/lib/python2.7/threading.py", line 754, in run
File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1136, in run_node
File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1282, in run_command_node
File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 656, in run
File "/usr/lib/python2.7/threading.py", line 585, in set
File "/usr/lib/python2.7/threading.py", line 407, in notifyAll
<type 'exceptions.TypeError'>: 'NoneType' object is not callable

• And sometimes this no failure output:

.
Done - 1 lines seen.

I also tested on Docker container Debian v9.4 using Python v2.7.13 and sarge v0.1.4: I get the same outputs.

When I kill (via [Ctrl]+[C]) one instance and immediately run the script again I get:

.Traceback (most recent call last):
  File "test_sarge_live.py", line 55, in <module>
    sys.exit(main())
  File "test_sarge_live.py", line 46, in main
    while(p.returncodes[0] is None):
IndexError: list index out of range

Done - 1 lines seen.
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
  File "/usr/lib/python2.7/threading.py", line 754, in run
  File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1136, in run_node
  File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1282, in run_command_node
  File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 656, in run
  File "/usr/lib/python2.7/threading.py", line 585, in set
  File "/usr/lib/python2.7/threading.py", line 407, in notifyAll
<type 'exceptions.TypeError'>: 'NoneType' object is not callable
[email protected]:/app# close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

but I think this is normal.

Is the tutorial code still accurate?

Original report by Dmitry Malinovsky (Bitbucket: malinoff, GitHub: malinoff).

Hi,

I found that I can't specify sleep n call before any other command (including another sleep n call) without waiting for it using async=True. Is it expected behavior?

This works as expected:

In [1]: from sarge import run

In [2]: p = run('echo abc | cat && sleep 5', async=True)
# Returns immediately
abc

In [3]: p.commands[-1].poll()

In [4]: p.commands[-1].poll()
Out[4]: 0

This is not:

In [5]: p = run('sleep 5 && echo abc | cat', async=True)
# Hangs for 5 seconds
abc

Original report by Dmitry Malinovsky (Bitbucket: malinoff, GitHub: malinoff).

Hi,

I tried to run a simple script: run_forever.py

import time

i = 0
while True:
    time.sleep(1)
    print(i)
    i += 1

with sarge as described in the docs:

In [1]: import sarge

In [2]: p = sarge.run('python run_forever.py', async=True, stdout=sarge.Capture())

However, I can't capture the output:

In [3]: p.stdout.readline()
Out[3]: b''

In [4]: p.commands[0].stdout.readline()
Out[4]: b''

Am I doing it wrong? How can I achieve that?

Original report by pahaz NA (Bitbucket: pahaz, GitHub: pahaz).

from sarge import shell_shlex, shell_format
import shlex
import subprocess

cmd = shell_format("python -c {0}", "print('aw')")
print(cmd)

This code produse:

C:\Users\pahaz_000>C:\Python27\python.exe z.py
python -c 'print('\''aw'\'')'

And on windows whis in incorrect:

C:\Users\pahaz_000\PycharmProjects\dokku>python -c 'print('\''aw'\'')'
  File "<string>", line 1
    'print('\''aw'\'')'
                      ^
SyntaxError: unexpected character after line continuation character

On linux correct:

(venv)[email protected]:/vagrant# python -c 'print('\''aw'\'')'
aw

Original report by Paul Moore (Bitbucket: pmoore, GitHub: pmoore).

The original report had attachments: yada.py, yadatest.py

I wanted to use sarge to read process output line by line. As a small test case I created the attached files, yada.py and yadatest.py. The yada.py script writes its argument repeatedly, with a user-specified delay between each write. Standard output is flushed after each write. Used at the command line, this works as expected.

The yadatest script exercises yada.py via sarge, capturing stdout and displaying each line as it is received. Again, stdout is flushed after each write.

The yadatest script runs for a while (about 5 seconds) with no output, and then fails with the following error:

#!python

>python .\yadatest.py
Traceback (most recent call last):
  File "yada.py", line 11, in <module>
    print(args.text, flush=True)
OSError: [Errno 22] Invalid argument
Exception OSError: OSError(22, 'Invalid argument') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'> ignored
0 foo

Did something close my child process' stdout? Note that there is no "Error encountered" message, implying that the failure was in the child, not in the parent.

Original report by Wes Turner (Bitbucket: westurner, GitHub: westurner).

Is there a (easy) way to raise an Exception if the returncode is not (by default?) 0?