An extremely configurable markdown reverser for Python3.

Overview

🔄 Unmarkd

codecov Code style: black CI PyPI - Downloads

A markdown reverser.


Unmarkd is a BeautifulSoup-powered Markdown reverser written in Python and for Python.

Why

This is created as a StackSearch (one of my other projects) dependency. In order to create a better API, I needed a way to reverse HTML. So I created this.

There are similar projects (written in Ruby) but I have not found any written in Python (or for Python) later I found a popular library, html2text. But Unmarkd still is still better. See comparison.

Installation

You know the drill

pip install unmarkd

Known issues

  • Nested lists are not properly indented (#4) Fixed in #11
  • Blockquote bug (#18) Fixed in #23

Comparison

TL;DR: Html2Text is fast. If you don't need much configuration, you could use Html2Text for the little speed increase.

Click to expand

Speed

TL;DR: Unmarkd < Html2Text

Html2Text is basically faster:

Benchmark

(The DOC variable used can be found here)

Unmarkd sacrifices speed for power.

Html2Text directly uses Python's html.parser module (in the standard library). On the other hand, Unmarkd uses the powerful HTML parsing library, beautifulsoup4. BeautifulSoup can be configured to use different HTML parsers. In Unmarkd, we configure it to use Python's html.parser, too.

But another layer of code means more code is ran.

I hope that's a good explanation of the speed difference.

Correctness

TL;DR: Unmarkd == Html2Text

I actually found two html-to-markdown libraries. One of them was Tomd which had an incorrect implementation:

Actual results

It seems to be abandoned, anyway.

Now with Html2Text and Unmarkd:

Epic showdown

In other words, they work

Configurability

TL;DR: Unmarkd > Html2Text

This is Unmarkd's strong point.

In Html2Text, you only have a limited set of options.

In Unmarkd, you can subclass the BaseUnmarker and implement conversions for new tags (e.g. ), etc. In my opinion, it's much easier to extend and configure Unmarkd.

Unmarkd was originally written as a StackSearch dependancy.

Html2Text has no options for configuring parsing of code blocks. Unmarkd does

Documentation

Here's an example of basic usage

I love markdown!")) # Output: **I *love* markdown!**">
import unmarkd
print(unmarkd.unmark("I love markdown!"))
# Output: **I *love* markdown!**

or something more complex (shamelessly taken from here):

Sample Markdown

This is some basic, sample markdown.

Second Heading

  • Unordered lists, and:
    1. One
    2. Two
    3. Three
  • More

Blockquote

And bold, italics, and even italics and later bold. Even strikethrough. A link to somewhere.

And code highlighting:

var foo = 'bar';

function baz(s) {
   return foo + ':' + s;
}

Or inline code like var foo = 'bar';.

Or an image of bears

bears

The end ...

""" print(unmarkd.unmark(html_doc))">
import unmarkd
html_doc = R"""

Sample Markdown

This is some basic, sample markdown.

Second Heading

  • Unordered lists, and:
    1. One
    2. Two
    3. Three
  • More

Blockquote

And bold, italics, and even italics and later bold. Even strikethrough. A link to somewhere.

And code highlighting:

var foo = 'bar';

function baz(s) {
   return foo + ':' + s;
}

Or inline code like var foo = 'bar';.

Or an image of bears

bears

The end ...

""" print(unmarkd.unmark(html_doc))

and the output:

    # Sample Markdown


    This is some basic, sample markdown.

    ## Second Heading



    - Unordered lists, and:
     1. One
     2. Two
     3. Three
    - More

    >Blockquote


    And **bold**, *italics*, and even *italics and later **bold***. Even ~~strikethrough~~. [A link](https://markdowntohtml.com) to somewhere.

    And code highlighting:


    ```js
    var foo = 'bar';

    function baz(s) {
       return foo + ':' + s;
    }
    ```


    Or inline code like `var foo = 'bar';`.

    Or an image of bears

    ![bears](http://placebear.com/200/200)

    The end ...

Extending

Brief Overview

Most functionality should be covered by the BasicUnmarker class defined in unmarkd.unmarkers.

If you need to reverse markdown from StackExchange (as in the case for my other project), you may use the StackOverflowUnmarker (or it's alias, StackExchangeUnmarker), which is also defined in unmarkd.unmarkers.

Customizing

If the above two classes do not suit your needs, you can subclass the unmarkd.unmarkers.BaseUnmarker abstract class.

Currently, you can optionally override the following methods:

  • detect_language (parameters: 1)
    • Parameters:
      • html: bs4.BeautifulSoup
    • When a fenced code block is approached, this function is called with a parameter of type bs4.BeautifulSoup passed to it; this is the element the code block was detected from (i.e. pre).
    • This function is responsible for detecting the programming language (or returning '' if none was detected) of the code block.
    • Note: This method is different from unmarkd.unmarkers.BasicUnmarker. It is simpler and does less checking/filtering

But Unmarkd is more flexible than that.

Customizable constants

There are currently 3 constants you may override:

  • Formats: NOTE: Use the Format String Syntax
    • UNORDERED_FORMAT
      • The string format of unordered (bulleted) lists.
    • ORDERED_FORMAT
      • The string format of ordered (numbered) lists.
  • Miscellaneous:
    • ESCAPABLES
      • A container (preferably a set) of length-1 str that should be escaped
Customize converting HTML tags

For an HTML tag some_tag, you can customize how it's converted to markdown by overriding a method like so:

from unmarkd.unmarkers import BaseUnmarker
class MyCustomUnmarker(BaseUnmarker):
    def tag_some_tag(self, child) -> str:
        ...  # parse code here

To reduce code duplication, if your tag also has aliases (e.g. strong is an alias for b in HTML) then you may modify the TAG_ALIASES.

If you really need to, you may also modify DEFAULT_TAG_ALIASES. Be warned: if you do so, you will also need to implement the aliases (currently em and strong).

Utility functions when overriding

You may use (when extending) the following functions:

  • __parse, 2 parameters:
    • html: bs4.BeautifulSoup
      • The html to unmark. This is used internally by the unmark method and is slightly faster.
    • escape: bool
      • Whether to escape the characters inside the string or not. Defaults to False.
  • escape: 1 parameter:
    • string: str
      • The string to escape and make markdown-safe
  • wrap: 2 parameters:
    • element: bs4.BeautifulSoup
      • The element to wrap.
    • around_with: str
      • The character to wrap the element around with. WILL NOT BE ESCPAED
  • And, of course, tag_* and detect_language.
Comments
  • Nested lists of same type don't work

    Nested lists of same type don't work

    Both unordered and ordered list don't work when nested of the same type:

    Two nested ordered lists

    HTML:

    <ol>
        <li>Top level 1</li>
        <li>Top level 2
            <ol>
                <li>A</li>
                <li>B</li>
                <li>C</li>
            </ol>
        </li>
        <li>Top level 3</li>
    </ol>
    

    Output:

    1. Top level 1
     2. Top level 2
            
     1. A
     2. B
     3. C
     3. Top level 3
    

    Two nested unordered lists

    HTML:

    <ul>
        <li>Top level 1</li>
        <li>Top level 2
            <ul>
                <li>A</li>
                <li>B</li>
                <li>C</li>
            </ul>
        </li>
        <li>Top level 3</li>
    </ul>
    

    Output:

    - Top level 1
    - Top level 2
            
    - A
    - B
    - C
    - Top level 3
    
    bug good first issue reproduced 
    opened by sirnacnud 3
  • [ImgBot] Optimize images

    [ImgBot] Optimize images

    Beep boop. Your images are optimized!

    Your image file size has been reduced by 39% 🎉

    Details

    | File | Before | After | Percent reduction | |:--|:--|:--|:--| | /assets/correct.png | 372.04kb | 224.67kb | 39.61% | | /assets/tomd_cant_handle.png | 347.74kb | 210.22kb | 39.55% | | /assets/benchmark.png | 219.28kb | 141.36kb | 35.53% | | | | | | | Total : | 939.06kb | 576.25kb | 38.64% |


    📝 docs | :octocat: repo | 🙋🏾 issues | 🏪 marketplace

    ~Imgbot - Part of Optimole family

    opened by imgbot[bot] 1
  • Fix indent getting added to list children that weren't other lists

    Fix indent getting added to list children that weren't other lists

    I was running in to an issue where list items using tags where getting indented when they shouldn't of been.

    Example:

    <ol>
        <li>A</li>
        <li>B</li>
        <li><b>C</b></li>
    </ol>
    

    Output:

    1. A
    2. B
    3.     **C**
    

    I added a test for this case as well. When doing the roundtrip style test, this indentation got lost, so I made the test compare the markdown output.

    opened by sirnacnud 1
  • Support for tables

    Support for tables

    While Unmarkd currently supports tables, it spits out the html it was given. It would be nice if it supported tables:

    | Syntax      | Description |
    | ----------- | ----------- |
    | Header      | Title       |
    | Paragraph   | Text        |
    
    enhancement 
    opened by ThatXliner 1
  • Nested lists are not properly indented

    Nested lists are not properly indented

    When the following HTML block is parsed:

    <ul>
        <li>Unordered lists, and:
            <ol>
                <li>One</li>
                <li>Two</li>
                <li>Three</li>
            </ol>
        </li>
        <li>More</li>
    </ul>
    

    The output is incorrect:

     * Unordered lists, and:
     0. One
     1. Two
     2. Three
     * More
    
    bug 
    opened by ThatXliner 1
  • Blockquote bug

    Blockquote bug

    Apply this patch:

    diff --git a/tests/test_roundtrip.py b/tests/test_roundtrip.py
    index a836024..5c1e097 100644
    --- a/tests/test_roundtrip.py
    +++ b/tests/test_roundtrip.py
    @@ -1,10 +1,9 @@
     import unicodedata
     
     import markdown_it
    -from hypothesis import assume, example, given
    -from hypothesis import strategies as st
    -
     import unmarkd
    +from hypothesis import assume, example, given, reproduce_failure
    +from hypothesis import strategies as st
     
     md = markdown_it.MarkdownIt()
     
    @@ -17,6 +16,7 @@ def helper(text: str, func=unmarkd.unmark) -> None:
     
     
     @given(text=st.text(st.characters(blacklist_categories=("Cc", "Cf", "Cs", "Co", "Cn"))))
    [email protected]_failure("6.10.1", b"AAEADgEADgEADgA=")
     def test_roundtrip_commonmark_unmark(text):
         assume(unicodedata.normalize("NFKC", text) == text)
         helper(text)
    
    
    
    

    Or add an example with text=">>>". Tests will fail

    bug 
    opened by ThatXliner 0
  • Update README for better comparison

    Update README for better comparison

    1. html2text is fast but not very configurable (there's only so any options)
    2. Tomd sucks
    3. Add an unmarker (with html2text-style configuration) to prove that unmarkd's configurability is at least equal to html2text
    documentation 
    opened by ThatXliner 0
  • Use a more reliable markdown parser

    Use a more reliable markdown parser

    Instead of using commonmark, maybe https://github.com/executablebooks/markdown-it-py, https://github.com/trentm/python-markdown2, https://github.com/lepture/mistune, or https://github.com/Python-Markdown/markdown.

    Also, I found tomd which might render this project useless 😬

    tests 
    opened by ThatXliner 0
  • Cannot handle nested bold and italics

    Cannot handle nested bold and italics

    When encountering input like <em>Italic and <strong>bold and italic</strong></em>, the output is wrong, usually shadowed by the outer tag (in this case, <em>)

    bug 
    opened by ThatXliner 0
  • Optimize code

    Optimize code

    I've noticed that unmarkers.BaseUnmarker been documented as an "abstract base class" when we're actually using it otherwise.

    Also, there's some dead code and we should actually sprinkle @staticmethod on some of them.

    Here's my idea:

    • Move all the tag_* methods in BaseUnmarker ➡️ BasicUnmarker
    • Rename: BaseUnmarker ➡️ AbstractUnmarker
    • Alias: BaseUnmarker ➡️ BasicMarker
    • Run shed on the whole codebase (with --refactor)

    Version bump: minor

    enhancement 
    opened by ThatXliner 0
  • Save CSS information

    Save CSS information

    1. Parse any css files or style tags found. Save it
    2. When a class attribute is found, try to resolve it to the css
    3. Add the resolved to the style attribute: convert to inline css
    enhancement 
    opened by ThatXliner 1
Releases(v0.1.9)
Owner
ThatXliner
I code Python. To me, programming is a logic puzzle. A fun one :D
ThatXliner
Convert Photoshop curves (acv) to xmp presets for Lightroom

acv2xmp Convert Photoshop curves (acv) to Lightroom preset (xmp) acv2xmp.py Basic command prompt that relies on standard library only and can be used

5 Feb 06, 2022
E-Paper display loop with plugins

PaperPi V3 NOTE This version of PaperPi is under heavy development and is not ready for the average user. We are working on adding more screen compati

Aaron Ciuffo 34 Dec 30, 2022
Repo Home WPDrawBot - (Repo, Home, WP) A powerful programmatic 2D drawing application for MacOS X which generates graphics from Python scripts. (graphics, dev, mac)

DrawBot DrawBot is a powerful, free application for macOS that invites you to write Python scripts to generate two-dimensional graphics. The built-in

Frederik Berlaen 342 Dec 27, 2022
Example applications, dashboards, scripts, notebooks, and other utilities built using Polygon.io

Polygon.io Examples Example applications, dashboards, scripts, notebooks, and other utilities built using Polygon.io. Examples Preview Name Type Langu

Tim Paine 4 Jun 01, 2022
B-Pkg is a simple tool in python for installing all basic package in termux

Basic-Pkg 👉🏻 Basic-Pkg 👈🏻 B-Pkg is a simple tool in python for installing all basic package in termux This is my first tool, I hope you will like

Macgaiver 3 Oct 21, 2021
A web UI for managing your 351ELEC device ROMs.

351ELEC WebUI A web UI for managing your 351ELEC device ROMs. Requirements Python 3 or Python 2.7 are required. If the ftfy package is installed, it w

Ben Phelps 5 Sep 26, 2022
Create an application to visualize single/multiple Xandar Kardian people counting sensors detection result for a indoor area.

Program Design Purpose: We want to create an application to visualize single/multiple Xandar Kardian people counting sensors detection result for a indoor area.

2 Dec 28, 2022
Projeto-menu - This project is designed to learn more about control mechanisms in Python programming

Projeto-menu - This project is designed to learn more about control mechanisms in Python programming

Henrik Ricarte 2 Mar 01, 2022
A python script that fetches the grades of a student from a WAEC result in pdf format.

About waec-result-analyzer A python script that fetches the grades of a student from a WAEC result in pdf format. Built for federal government college

Oshodi Kolapo 2 Dec 04, 2021
Polypheny Connector for Python

Polypheny Connector for Python This enables Python programs to access Polypheny databases, using an API that is compliant with the Python Database API

Polypheny 3 Jan 03, 2022
Better firefox bookmarks script for rofi

rofi-bookmarks Small python script to open firefox bookmarks with rofi. Features Icons! Only show bookmarks in a specified bookmark folder Show entire

32 Nov 10, 2022
a simple proof system I made to learn math without any mistakes

math_up a simple proof system I made to learn math without any mistakes 0. Short Introduction test yourself, enjoy your math! math_up is an NBG-based,

양현우 5 Jun 04, 2021
Windows symbol tables for Volatility 3

Windows Symbol Tables for Volatility 3 This repository is the Windows Symbol Table storage for Volatility 3. How to Use $ git clone https://github.com

JPCERT Coordination Center 31 Dec 25, 2022
Handwrite - Type in your Handwriting!

Handwrite - Type in your Handwriting! Ever had those long-winded assignments, that the teacher always wants handwritten?

coded 7 Dec 06, 2022
Purge all transformation orientations addon for Blender 2.8 and newer versions

CTO Purge This add-on adds a new button to Blender's Transformation Orientation panel which empowers the user to purge all of his/her custom transform

MMMrqs 10 Dec 29, 2022
Tracking stock volatility.

SP500-highlow-tracking Track stock volatility. Being a useful indicator of the stock price volatility, High-Low gap represents the price range of the

Thong Huynh 13 Sep 07, 2022
This is a Blender 2.9 script for importing mixamo Models to Godot-3

Mixamo-To-Godot This is a Blender 2.9 script for importing mixamo Models to Godot-3 The script does the following things Imports the mixamo models fro

8 Sep 02, 2022
WordPress-style shortcodes for Python

Python Shortcodes WordPress-style shortcodes for Python Create and use WordPress-style shortcodes in your Python based app. Example # static output de

Bob 1 Dec 22, 2021
Web app to find your chance of winning at Texas Hold 'Em

poker_mc Web app to find your chance of winning at Texas Hold 'Em A working version of this project is deployed at poker-mc.ue.r.appspot.com. It's run

Aadith Vittala 7 Sep 15, 2021
This repo is for scripts to run various clients at the merge f2f

merge-f2f This repo is for scripts to run various clients at the merge f2f. Tested with Lighthouse! Tested with Geth! General dependecies sudo apt-get

Parithosh Jayanthi 2 Apr 03, 2022