Shell scripts made simple 🐚

Last update: Dec 27, 2022

Related tags

Overview

zxpy

Shell scripts made simple 🐚

Inspired by Google's zx, but made much simpler and more accessible using Python.

Rationale

Bash is cool, and it's extremely powerful when paired with linux coreutils and pipes. But apart from that, it's a whole another language to learn, and has a (comparatively) unintuitive syntax for things like conditionals and loops.

zxpy aims to supercharge bash by allowing you to write scripts in Python, but with native support for bash commands and pipes:

#! /usr/bin/env zxpy
todo_comments = ~"git grep -n TODO"
for todo in todo_comments.splitlines():
    filename, lineno, code = todo.split(':', 2)
    *_, comment = code.partition('TODO')
    print(f"{filename:<40} on line {lineno:<4}: {comment.lstrip(': ')}")

Running this, we get:

$ ./todo_check.py
README.md                                on line 154 : move this content somewhere more sensible.
instachat/lib/models/message.dart        on line 7   : rename to uuid
instachat/lib/models/update.dart         on line 13  : make int
instachat/lib/services/chat_service.dart on line 211 : error handling
server/api/api.go                        on line 94  : move these to /chat/@:address
server/api/user.go                       on line 80  : check for errors instead of relying on zero value

Installation

pip install zxpy

Example

Make a file script.py (The name and extension can be anything):

#! /usr/bin/env zxpy
~'echo Hello world!'

file_count = ~'ls -1 | wc -l'
print("file count is:", file_count)

And then run it:

$ chmod +x ./script.py

$ ./script.py
Hello world!
file count is: 3

Run >>> help('zx') in Python REPL to find out more ways to use zxpy.

A more involved example: run_all_tests.py

#! /usr/bin/env zxpy
test_files = (~"find -name '*_test\.py'").splitlines()

for filename in test_files:
    try:
        print(f'Running {filename:.<50}', end='')
        output = ~f'python {filename}'  # variables in your shell commands :D
        assert output == ''
        print('Test passed!')
    except:
        print(f'Test failed.')

Output:

$ ./run_all_tests.py
Running ./tests/python_version_test.py....................Test failed.
Running ./tests/platform_test.py..........................Test passed!
Running ./tests/imports_test.py...........................Test passed!

Examples are all in the examples folder.

Interactive mode

$ zxpy
zxpy shell
Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0]

>>> ~"ls | grep '\.py'"
__main__.py
setup.py
zx.py
>>>

Also works with path/to/python -m zx

It can also be used to start a zxpy session in an already running REPL. Simply do:

>>> import zx; zx.start()

and zxpy should be enabled in the existing session.

Comments

shell hangs when `echo` is used

So for the following script

#! /usr/bin/env zxpy

~'echo Hello'

once I executed it, the shell hangs after printing Hello. When I press ctrl+c, I get:

^CTraceback (most recent call last):
  File "/usr/local/bin/zxpy", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/zx.py", line 59, in cli
    run_zxpy(filename, module)
  File "/usr/local/lib/python3.7/site-packages/zx.py", line 136, in run_zxpy
    "$shlex_quote": shlex.quote,
  File "./bug.py", line 3, in <module>
    ~'echo Hello'
  File "/usr/local/lib/python3.7/site-packages/zx.py", line 100, in run_shell_print
    sys.stdout.buffer.write(text)
KeyboardInterrupt

opened by hacker-DOM 17

Problems using a package

Hi there, the following simple script doesn't work

#!/usr/bin/env zxpy

import toml

def main():
    toml_string = """
[test]
x = "something"
"""
    parsed_toml = toml.loads(toml_string)
    print(parsed_toml)


if __name__ == '__main__':
    main()

I get

Traceback (most recent call last):
  File "/home/manfred/.local/bin/zxpy", line 8, in <module>
    sys.exit(cli())
  File "/home/manfred/.local/lib/python3.8/site-packages/zx.py", line 56, in cli
    run_zxpy(filename, module)
  File "/home/manfred/.local/lib/python3.8/site-packages/zx.py", line 121, in run_zxpy
    exec(code, None, {'__name__': '__main__'})
  File "./x.py", line 15, in <module>
    main()
  File "./x.py", line 10, in main
    parsed_toml = toml.loads(toml_string)
NameError: name 'toml' is not defined

opened by manfredlotz 14

Parameter for script

In a bash or python script I could have parameters when calling it. This seems to be impossible when using zxpy.

Example:

myscript /home

gives

Traceback (most recent call last):
  File "/home/manfred/.local/bin/zxpy", line 8, in <module>
    sys.exit(cli())
  File "/home/manfred/.local/lib/python3.8/site-packages/zx.py", line 54, in cli
    with open(filename) as file:
IsADirectoryError: [Errno 21] Is a directory: '/home'

Is there a way I just overlooked?

opened by manfredlotz 9

Documenting what syntax works and what doesn't

In the zxpy repl I tried various ways of issuing a cmd and some did not work.

zxpy shell
Python 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0]

>>> ~'uname'
Linux
>>> cmd = 'uname'
>>> ~f'{cmd}'
Linux
>>> ~cmd
Traceback (most recent call last):
  File "/home/manfred/.local/lib/python3.8/site-packages/zx.py", line 295, in install
    exec(code_obj)
  File "<input>", line 1, in <module>
TypeError: bad operand type for unary ~: 'str'
>>> cmd = 'uname -a'
>>> ~f'{cmd}'
/bin/sh: 1: uname -a: not found
>>> ~cmd
Traceback (most recent call last):
  File "/home/manfred/.local/lib/python3.8/site-packages/zx.py", line 295, in install
    exec(code_obj)
  File "<input>", line 1, in <module>
TypeError: bad operand type for unary ~: 'str'

opened by manfredlotz 8

How to deal with return codes

Perhaps I miss something easy.

But I could not see how I would deal with return codes > 0, i.e a command I issue returns an error. Example: ~'cat /etc/shadow

opened by manfredlotz 7

NameError: name 'run_shell' is not defined

Sorry for being a pain in the neck.

But, now using 1.4.2 it seems something else is broken

Running examples/script.py I get

Traceback (most recent call last):
  File "/home/manfred/.local/bin/zxpy", line 8, in <module>
    sys.exit(cli())
  File "/home/manfred/.local/lib/python3.8/site-packages/zx.py", line 56, in cli
    run_zxpy(filename, module)
  File "/home/manfred/.local/lib/python3.8/site-packages/zx.py", line 121, in run_zxpy
    exec(code, {'__name__': '__main__'})
  File "./y.py", line 2, in <module>
    ~'echo Hello world!'
NameError: name 'run_shell' is not defined

opened by manfredlotz 6

~ commands not working in a for loop

The following example

#!/usr/bin/env zxpy

def main():
    ~'/bin/echo "This works fine"'
    for u in [1, 2]:
        print(u)
        ~'/bin/echo "Not working"'

if __name__ == '__main__':
    main()

gives as output only:

This works fine
1
2

opened by manfredlotz 5

No live stdout like in zx

zx displays live output from each command for example:

#!/usr/bin/env zx
await $`sudo dnf update`

zx test.mjs
$ sudo dnf update
keybase                                          28 kB/s | 3.3 kB     00:00    
Dependencies resolved.
================================================================================
 Package                    Arch     Version                   Repository  Size
================================================================================
Installing:
 kernel                     x86_64   5.11.18-200.fc33          updates    152 k
 kernel-core                x86_64   5.11.18-200.fc33          updates     34 M
 kernel-devel               x86_64   5.11.18-200.fc33          updates     14 M
 kernel-modules             x86_64   5.11.18-200.fc33          updates     31 M
 kernel-modules-extra       x86_64   5.11.18-200.fc33          updates    2.1 M
Upgrading:
 bind-libs                  x86_64   32:9.11.31-1.fc33         updates     90 k
 bind-libs-lite             x86_64   32:9.11.31-1.fc33         updates    1.1 M
 bind-license               noarch   32:9.11.31-1.fc33         updates     17 k
 bind-utils                 x86_64   32:9.11.31-1.fc33         updates    238 k
 containers-common          noarch   4:1-16.fc33               updates     59 k
 copy-jdk-configs           noarch   4.0-0.fc33                updates     27 k
 cups                       x86_64   1:2.3.3op2-5.fc33         updates    1.3 M
 cups-client                x86_64   1:2.3.3op2-5.fc33         updates     72 k
 cups-filesystem            noarch   1:2.3.3op2-5.fc33         updates     14 k
 cups-ipptool               x86_64   1:2.3.3op2-5.fc33         updates    3.9 M
 cups-libs                  x86_64   1:2.3.3op2-5.fc33         updates    275 k
 ethtool                    x86_64   2:5.12-1.fc33             updates    216 k
 fmt                        x86_64   7.0.3-2.fc33              updates     88 k
 git                        x86_64   2.31.1-3.fc33             updates    122 k
 git-core                   x86_64   2.31.1-3.fc33             updates    3.6 M
 git-core-doc               noarch   2.31.1-3.fc33             updates    2.3 M
 git-credential-libsecret   x86_64   2.31.1-3.fc33             updates     21 k
 git-gui                    noarch   2.31.1-3.fc33             updates    250 k
 gitk                       noarch   2.31.1-3.fc33             updates    164 k
 gnome-online-accounts      x86_64   3.38.2-1.fc33             updates    479 k
 hwdata                     noarch   0.347-1.fc33              updates    1.5 M
 ibus-typing-booster        noarch   2.11.4-1.fc33             updates    902 k
 libnfsidmap                x86_64   1:2.5.3-2.fc33            updates     61 k
 libopenmpt                 x86_64   0.4.20-1.fc33             updates    553 k
 libxcrypt                  x86_64   4.4.20-2.fc33             updates    119 k
 libxcrypt-compat           x86_64   4.4.20-2.fc33             updates     91 k
 libxcrypt-devel            x86_64   4.4.20-2.fc33             updates     29 k
 nfs-utils                  x86_64   1:2.5.3-2.fc33            updates    419 k
 perl-Git                   noarch   2.31.1-3.fc33             updates     44 k
 podman                     x86_64   2:3.2.0-0.1.rc1.fc33      updates     12 M
 podman-plugins             x86_64   2:3.2.0-0.1.rc1.fc33      updates    1.3 M
 python3-babel              noarch   2.8.1-2.fc33              updates    5.7 M
 selinux-policy             noarch   3.14.6-37.fc33            updates     68 k
 selinux-policy-targeted    noarch   3.14.6-37.fc33            updates    8.0 M
 vim-filesystem             noarch   2:8.2.2825-1.fc33         updates     23 k
 vim-minimal                x86_64   2:8.2.2825-1.fc33         updates    695 k
 xdg-desktop-portal         x86_64   1.8.1-2.fc33              updates    354 k
 xdg-desktop-portal-devel   x86_64   1.8.1-2.fc33              updates    8.9 k
 xdg-utils                  noarch   1.1.3-9.fc33              updates     72 k
 zchunk-libs                x86_64   1.1.11-1.fc33             updates     46 k
Removing:
 kernel                     x86_64   5.11.15-200.fc33          @updates     0  
 kernel-core                x86_64   5.11.15-200.fc33          @updates    74 M
 kernel-devel               x86_64   5.11.15-200.fc33          @updates    56 M
 kernel-modules             x86_64   5.11.15-200.fc33          @updates    30 M
 kernel-modules-extra       x86_64   5.11.15-200.fc33          @updates   1.9 M

Transaction Summary
================================================================================
Install   5 Packages
Upgrade  40 Packages
Remove    5 Packages
Skip      1 Package

Total download size: 127 M

This is not working with zxpy since it's using simple subprocess to execute command and then returns output and only if there is no interactive prompt at the command.

For example

#! /usr/bin/env zxpy
~'sudo dnf update'

Will just sit there forever without any output.

opened by JayDoubleu 4

Different sys.argv behavour
Problem description

When I run a zxpy script, the behaviour of sys.argv is not the same as regular python.

Example:

Setup

I ran poetry init in an empy directory, and used poetry add zxpy to install zxpy (version 1.2.4).

python3

I have the following script (test_python.py):

#!/usr/bin/env python3 import sys print(sys.argv)

If I run the script as poetry run python3 ./test_python.py, the output is:

['./test_python.py']

If I then run chmod +x ./test_python.py && poetry run ./test_python.py, the output is:

['./test_python.py']

zxpy

I have the following script (test_zxpy.py):

#!/usr/bin/env zxpy import sys print(sys.argv)

If I run the script as poetry run zxpy ./test_zxpy.py, the output is:

['/home/techhazard/.cache/pypoetry/virtualenvs/aoeuaoe-BGFxEDCR-py3.9/bin/zxpy', 'test_zxpy.py']

If I then run chmod +x ./test_zxpy.py && poetry run ./test_zxpy.py, the output is:

['/home/techhazard/.cache/pypoetry/virtualenvs/aoeuaoe-BGFxEDCR-py3.9/bin/zxpy', 'test_zxpy.py']

Expected behaviour:

When zxpy is run, the sys.argv values should not include the zxpy executable.
opened by techhazard 3

unary operator syntax doesn't work with print()

Code:

#! /usr/bin/env zxpy

var = "test string"

print(~f"echo {var}")

Output:

$ ./simplescript.py
Traceback (most recent call last):
  File "/home/ubuntu/zxpy-test/venv/bin/zxpy", line 8, in <module>
    sys.exit(cli())
  File "/home/ubuntu/zxpy-test/venv/lib/python3.8/site-packages/zx.py", line 55, in cli
    run_zxpy(filename, module)
  File "/home/ubuntu/zxpy-test/venv/lib/python3.8/site-packages/zx.py", line 85, in run_zxpy
    exec(compile(module, filename, mode='exec'))
  File "./simplescript.py", line 5, in <module>
    print(~f"echo {var}")
TypeError: bad operand type for unary ~: 'str'

zxpy version: 1.2.3

opened by Jackenmen 3

bad operand type for unary ~: 'str'

This is my code for find some zip in dir but it cause the exception below l = (~f'find -maxdepth 1 -name "*{x}*.zip" -type f').splitlines()

Traceback (most recent call last):
 File "/usr/local/bin/zxpy", line 8, in <module>
   sys.exit(cli())
 File "/usr/local/lib/python3.8/dist-packages/zx.py", line 55, in cli
   run_zxpy(filename, module)
 File "/usr/local/lib/python3.8/dist-packages/zx.py", line 85, in run_zxpy
   exec(compile(module, filename, mode='exec'))
 File "./test.py", line 21, in <module>
   l = (~f'find -maxdepth 1 -name "*{x}*.zip" -type f').splitlines()
TypeError: bad operand type for unary ~: 'str'

when i split this command into two line, and all is fine

l = ~f'find -maxdepth 1 -name "*{x}*.zip" -type f'
l = l.splitlines()

By the way, data = (~'ls some_dir/').splitlines() this command is ok. So maybe it only occurs on format string?

Thanks a lot

Ubuntu 20.04.2 LTS AArch64 Python 3.8.5 zxpy 1.2.1

opened by tjjh89017 3

Code cleanup
[ ] #35

[x] Fix unused import in tests

[x] Fix Python 3.6/3.7 support (use pytest-typing-imports)

[x] Remove __main__.py

[x] Sort gitignore

[x] Setup black

[x] Simple refactors (like sys.argv)
opened by tusharsadhwani 0

Releases(1.6.2)

1.6.2(Aug 6, 2022)
What's Changed

Replace the custom interpreter with a builtin one by @tusharsadhwani in https://github.com/tusharsadhwani/zxpy/pull/49

Full Changelog: https://github.com/tusharsadhwani/zxpy/compare/1.6.1...1.6.2
Source code(tar.gz)
Source code(zip)
zxpy-1.6.2-py3-none-any.whl(7.82 KB)
1.6.1(May 8, 2022)
Fix bug in interactive mode invocation: zxpy by itself wouldn't work before this.

Source code(tar.gz)
Source code(zip)
zxpy-1.6.1-py3-none-any.whl(7.85 KB)
1.6.0(Mar 7, 2022)

Added interactive mode option: zxpy -i myfile.py.

Full Changelog: https://github.com/tusharsadhwani/zxpy/compare/1.5.2...1.6.0
Source code(tar.gz)
Source code(zip)
zxpy-1.6.0-py3-none-any.whl(7.89 KB)
1.5.2(Oct 19, 2021)
What's Changed

Break Multiline REPL Statements by @udasitharani in https://github.com/tusharsadhwani/zxpy/pull/40

New Contributors

@udasitharani made their first contribution in https://github.com/tusharsadhwani/zxpy/pull/40

Full Changelog: https://github.com/tusharsadhwani/zxpy/compare/1.5.1...1.5.2
Source code(tar.gz)
Source code(zip)
zxpy-1.5.2-py3-none-any.whl(7.63 KB)
1.5.1(Oct 18, 2021)
use read1() instead of select polling, to simplify some logic

remove the use of echo -n, as that is linux-specific

Source code(tar.gz)
Source code(zip)
zxpy-1.5.1-py3-none-any.whl(7.63 KB)
1.5.0(Oct 11, 2021)

Adds Python3.10 support
Source code(tar.gz)
Source code(zip)
zxpy-1.5.0-py3-none-any.whl(7.58 KB)
1.4.7(Sep 29, 2021)
Use the select module to do non-blocking reads from the shell process, instead of reading 1 byte at a time. Leads to significantly faster output for large streams, without any blocking.

Use process.communicate() to read stdout and stderr without causing deadlock due to buffer overflow.

Source code(tar.gz)
Source code(zip)
zxpy-1.4.7-py3-none-any.whl(7.45 KB)
1.4.6(Sep 28, 2021)
Ensure the package actually works in Python 3.6+

Make sure that ~'...' raises an error if the process exits with a non-zero return code

Fix an edge case in supported syntax: [a, b, c] = ~'...'

Source code(tar.gz)
Source code(zip)
zxpy-1.4.6-py3-none-any.whl(7.19 KB)
1.4.5(Aug 3, 2021)
Now you can pass whole commands as variables inside f-strings, without quoting.

Take this piece of code:

>>> cmd = 'uname -a' >>> ~f'{cmd}' /bin/sh: 1: uname -a: not found

This is because uname -a was quoted into 'uname -a' to avoid shell injection.

To avoid this, support for :raw format_spec as added:

>>> cmd = 'uname -a' >>> ~f'{cmd:raw}' Linux pop-os 5.11.0 [...] x86_64 GNU/Linux

Note that this shouldn't be used with external data, or this will expose you to shell injection.
Source code(tar.gz)
Source code(zip)
zxpy-1.4.5-py3-none-any.whl(6.68 KB)
1.4.4(Jul 29, 2021)

Previously, only a few select ~'...' statements were being printed out, rather than any bash-string without any assignments to its left. This patch version fixes printing logic.
Source code(tar.gz)
Source code(zip)
1.4.3(Jul 17, 2021)

Fixes the previous broken build, refer #19
Source code(tar.gz)
Source code(zip)
zxpy-1.4.3-py3-none-any.whl(6.69 KB)
1.4.2(Jul 16, 2021)

Refer #18
Source code(tar.gz)
Source code(zip)
zxpy-1.4.2-py3-none-any.whl(6.70 KB)
1.4.1(Jul 10, 2021)
__name__ will now be '__main__', just like how it is when running any other file directly

Passing additional arguments to zxpy command is now supported

Source code(tar.gz)
Source code(zip)
zxpy-1.4.1-py3-none-any.whl(6.70 KB)
1.4.0(Jul 2, 2021)

Anything passed as an f-string into a bash string will be automatically quoted with shlex.quote.

i.e., ~f'{xyz}' is the same as ~f'{shlex.quote(xyz)}
Source code(tar.gz)
Source code(zip)
zxpy-1.4.0-py3-none-any.whl(6.52 KB)
1.3.0(Jun 23, 2021)
Alternate method of invoking shell commands: stdout, stderr, return_code = ~'...'. Fixes #6

Add a few tests.

Source code(tar.gz)
Source code(zip)
zxpy-1.3.0-py3-none-any.whl(6.16 KB)
1.2.5(Jun 9, 2021)
Removes zxpy executable from sys.argv, to mimic default python behaviour (#10)

Source code(tar.gz)
Source code(zip)
zxpy-1.2.5-py3-none-any.whl(5.73 KB)
1.2.4(May 21, 2021)
Finally figured out the correct way to handle expr transformations. Cleaned up a bunch of AST code.

Shell strings and shell f-strings should be usable inside call statements now, eg. print(~'echo test')

Source code(tar.gz)
Source code(zip)
zxpy-1.2.4-py3-none-any.whl(5.71 KB)
1.2.3(May 21, 2021)
Renamed zx.start() to zx.install() for clarity

Source code(tar.gz)
Source code(zip)
zxpy-1.2.3-py3-none-any.whl(5.74 KB)
1.2.2(May 16, 2021)
Fix using string attributes over bash f-strings, i.e. (~f'...').splitlines()

Source code(tar.gz)
Source code(zip)
zxpy-1.2.2-py3-none-any.whl(5.74 KB)
1.2.1(May 12, 2021)

zx.start() in REPL now preserves the REPL's imports and locals
Source code(tar.gz)
Source code(zip)
zxpy-1.2.1-py3-none-any.whl(5.74 KB)
1.2.0(May 10, 2021)
Improve support for shell strings inside functions

Allow accessing attributes on shell strings, like (~"grep ...").splitlines()

Source code(tar.gz)
Source code(zip)
zxpy-1.2.0-py3-none-any.whl(5.64 KB)
1.1.1(May 10, 2021)

Remove the use of := operator for backwards compatibility.
Source code(tar.gz)
Source code(zip)
zxpy-1.1.1-py3-none-any.whl(5.49 KB)
1.1.0(May 10, 2021)
It no longer waits for the shell command to finish before printing its output. eg. ~'echo hello; sleep 1; echo bye'

Added REPL support: Running zxpy with no arguments starts up a REPL. More info here.

Source code(tar.gz)
Source code(zip)
zxpy-1.1.0-py3-none-any.whl(4.88 KB)
1.0.1(May 9, 2021)

Simple improvements to the code quality and documentation.
Source code(tar.gz)
Source code(zip)
zxpy-1.0.1-py3-none-any.whl(3.59 KB)
1.0.0(May 8, 2021)

Source code(tar.gz)
Source code(zip)
zxpy-1.0.0-py3-none-any.whl(3.47 KB)