An extremely configurable markdown reverser for Python3.

Overview

🔄 Unmarkd

codecov Code style: black CI PyPI - Downloads

A markdown reverser.


Unmarkd is a BeautifulSoup-powered Markdown reverser written in Python and for Python.

Why

This is created as a StackSearch (one of my other projects) dependency. In order to create a better API, I needed a way to reverse HTML. So I created this.

There are similar projects (written in Ruby) but I have not found any written in Python (or for Python) later I found a popular library, html2text. But Unmarkd still is still better. See comparison.

Installation

You know the drill

pip install unmarkd

Known issues

  • Nested lists are not properly indented (#4) Fixed in #11
  • Blockquote bug (#18) Fixed in #23

Comparison

TL;DR: Html2Text is fast. If you don't need much configuration, you could use Html2Text for the little speed increase.

Click to expand

Speed

TL;DR: Unmarkd < Html2Text

Html2Text is basically faster:

Benchmark

(The DOC variable used can be found here)

Unmarkd sacrifices speed for power.

Html2Text directly uses Python's html.parser module (in the standard library). On the other hand, Unmarkd uses the powerful HTML parsing library, beautifulsoup4. BeautifulSoup can be configured to use different HTML parsers. In Unmarkd, we configure it to use Python's html.parser, too.

But another layer of code means more code is ran.

I hope that's a good explanation of the speed difference.

Correctness

TL;DR: Unmarkd == Html2Text

I actually found two html-to-markdown libraries. One of them was Tomd which had an incorrect implementation:

Actual results

It seems to be abandoned, anyway.

Now with Html2Text and Unmarkd:

Epic showdown

In other words, they work

Configurability

TL;DR: Unmarkd > Html2Text

This is Unmarkd's strong point.

In Html2Text, you only have a limited set of options.

In Unmarkd, you can subclass the BaseUnmarker and implement conversions for new tags (e.g. ), etc. In my opinion, it's much easier to extend and configure Unmarkd.

Unmarkd was originally written as a StackSearch dependancy.

Html2Text has no options for configuring parsing of code blocks. Unmarkd does

Documentation

Here's an example of basic usage

I love markdown!")) # Output: **I *love* markdown!**">
import unmarkd
print(unmarkd.unmark("I love markdown!"))
# Output: **I *love* markdown!**

or something more complex (shamelessly taken from here):

Sample Markdown

This is some basic, sample markdown.

Second Heading

  • Unordered lists, and:
    1. One
    2. Two
    3. Three
  • More

Blockquote

And bold, italics, and even italics and later bold. Even strikethrough. A link to somewhere.

And code highlighting:

var foo = 'bar';

function baz(s) {
   return foo + ':' + s;
}

Or inline code like var foo = 'bar';.

Or an image of bears

bears

The end ...

""" print(unmarkd.unmark(html_doc))">
import unmarkd
html_doc = R"""

Sample Markdown

This is some basic, sample markdown.

Second Heading

  • Unordered lists, and:
    1. One
    2. Two
    3. Three
  • More

Blockquote

And bold, italics, and even italics and later bold. Even strikethrough. A link to somewhere.

And code highlighting:

var foo = 'bar';

function baz(s) {
   return foo + ':' + s;
}

Or inline code like var foo = 'bar';.

Or an image of bears

bears

The end ...

""" print(unmarkd.unmark(html_doc))

and the output:

    # Sample Markdown


    This is some basic, sample markdown.

    ## Second Heading



    - Unordered lists, and:
     1. One
     2. Two
     3. Three
    - More

    >Blockquote


    And **bold**, *italics*, and even *italics and later **bold***. Even ~~strikethrough~~. [A link](https://markdowntohtml.com) to somewhere.

    And code highlighting:


    ```js
    var foo = 'bar';

    function baz(s) {
       return foo + ':' + s;
    }
    ```


    Or inline code like `var foo = 'bar';`.

    Or an image of bears

    ![bears](http://placebear.com/200/200)

    The end ...

Extending

Brief Overview

Most functionality should be covered by the BasicUnmarker class defined in unmarkd.unmarkers.

If you need to reverse markdown from StackExchange (as in the case for my other project), you may use the StackOverflowUnmarker (or it's alias, StackExchangeUnmarker), which is also defined in unmarkd.unmarkers.

Customizing

If the above two classes do not suit your needs, you can subclass the unmarkd.unmarkers.BaseUnmarker abstract class.

Currently, you can optionally override the following methods:

  • detect_language (parameters: 1)
    • Parameters:
      • html: bs4.BeautifulSoup
    • When a fenced code block is approached, this function is called with a parameter of type bs4.BeautifulSoup passed to it; this is the element the code block was detected from (i.e. pre).
    • This function is responsible for detecting the programming language (or returning '' if none was detected) of the code block.
    • Note: This method is different from unmarkd.unmarkers.BasicUnmarker. It is simpler and does less checking/filtering

But Unmarkd is more flexible than that.

Customizable constants

There are currently 3 constants you may override:

  • Formats: NOTE: Use the Format String Syntax
    • UNORDERED_FORMAT
      • The string format of unordered (bulleted) lists.
    • ORDERED_FORMAT
      • The string format of ordered (numbered) lists.
  • Miscellaneous:
    • ESCAPABLES
      • A container (preferably a set) of length-1 str that should be escaped
Customize converting HTML tags

For an HTML tag some_tag, you can customize how it's converted to markdown by overriding a method like so:

from unmarkd.unmarkers import BaseUnmarker
class MyCustomUnmarker(BaseUnmarker):
    def tag_some_tag(self, child) -> str:
        ...  # parse code here

To reduce code duplication, if your tag also has aliases (e.g. strong is an alias for b in HTML) then you may modify the TAG_ALIASES.

If you really need to, you may also modify DEFAULT_TAG_ALIASES. Be warned: if you do so, you will also need to implement the aliases (currently em and strong).

Utility functions when overriding

You may use (when extending) the following functions:

  • __parse, 2 parameters:
    • html: bs4.BeautifulSoup
      • The html to unmark. This is used internally by the unmark method and is slightly faster.
    • escape: bool
      • Whether to escape the characters inside the string or not. Defaults to False.
  • escape: 1 parameter:
    • string: str
      • The string to escape and make markdown-safe
  • wrap: 2 parameters:
    • element: bs4.BeautifulSoup
      • The element to wrap.
    • around_with: str
      • The character to wrap the element around with. WILL NOT BE ESCPAED
  • And, of course, tag_* and detect_language.
Comments
  • Nested lists of same type don't work

    Nested lists of same type don't work

    Both unordered and ordered list don't work when nested of the same type:

    Two nested ordered lists

    HTML:

    <ol>
        <li>Top level 1</li>
        <li>Top level 2
            <ol>
                <li>A</li>
                <li>B</li>
                <li>C</li>
            </ol>
        </li>
        <li>Top level 3</li>
    </ol>
    

    Output:

    1. Top level 1
     2. Top level 2
            
     1. A
     2. B
     3. C
     3. Top level 3
    

    Two nested unordered lists

    HTML:

    <ul>
        <li>Top level 1</li>
        <li>Top level 2
            <ul>
                <li>A</li>
                <li>B</li>
                <li>C</li>
            </ul>
        </li>
        <li>Top level 3</li>
    </ul>
    

    Output:

    - Top level 1
    - Top level 2
            
    - A
    - B
    - C
    - Top level 3
    
    bug good first issue reproduced 
    opened by sirnacnud 3
  • [ImgBot] Optimize images

    [ImgBot] Optimize images

    Beep boop. Your images are optimized!

    Your image file size has been reduced by 39% 🎉

    Details

    | File | Before | After | Percent reduction | |:--|:--|:--|:--| | /assets/correct.png | 372.04kb | 224.67kb | 39.61% | | /assets/tomd_cant_handle.png | 347.74kb | 210.22kb | 39.55% | | /assets/benchmark.png | 219.28kb | 141.36kb | 35.53% | | | | | | | Total : | 939.06kb | 576.25kb | 38.64% |


    📝 docs | :octocat: repo | 🙋🏾 issues | 🏪 marketplace

    ~Imgbot - Part of Optimole family

    opened by imgbot[bot] 1
  • Fix indent getting added to list children that weren't other lists

    Fix indent getting added to list children that weren't other lists

    I was running in to an issue where list items using tags where getting indented when they shouldn't of been.

    Example:

    <ol>
        <li>A</li>
        <li>B</li>
        <li><b>C</b></li>
    </ol>
    

    Output:

    1. A
    2. B
    3.     **C**
    

    I added a test for this case as well. When doing the roundtrip style test, this indentation got lost, so I made the test compare the markdown output.

    opened by sirnacnud 1
  • Support for tables

    Support for tables

    While Unmarkd currently supports tables, it spits out the html it was given. It would be nice if it supported tables:

    | Syntax      | Description |
    | ----------- | ----------- |
    | Header      | Title       |
    | Paragraph   | Text        |
    
    enhancement 
    opened by ThatXliner 1
  • Nested lists are not properly indented

    Nested lists are not properly indented

    When the following HTML block is parsed:

    <ul>
        <li>Unordered lists, and:
            <ol>
                <li>One</li>
                <li>Two</li>
                <li>Three</li>
            </ol>
        </li>
        <li>More</li>
    </ul>
    

    The output is incorrect:

     * Unordered lists, and:
     0. One
     1. Two
     2. Three
     * More
    
    bug 
    opened by ThatXliner 1
  • Blockquote bug

    Blockquote bug

    Apply this patch:

    diff --git a/tests/test_roundtrip.py b/tests/test_roundtrip.py
    index a836024..5c1e097 100644
    --- a/tests/test_roundtrip.py
    +++ b/tests/test_roundtrip.py
    @@ -1,10 +1,9 @@
     import unicodedata
     
     import markdown_it
    -from hypothesis import assume, example, given
    -from hypothesis import strategies as st
    -
     import unmarkd
    +from hypothesis import assume, example, given, reproduce_failure
    +from hypothesis import strategies as st
     
     md = markdown_it.MarkdownIt()
     
    @@ -17,6 +16,7 @@ def helper(text: str, func=unmarkd.unmark) -> None:
     
     
     @given(text=st.text(st.characters(blacklist_categories=("Cc", "Cf", "Cs", "Co", "Cn"))))
    [email protected]_failure("6.10.1", b"AAEADgEADgEADgA=")
     def test_roundtrip_commonmark_unmark(text):
         assume(unicodedata.normalize("NFKC", text) == text)
         helper(text)
    
    
    
    

    Or add an example with text=">>>". Tests will fail

    bug 
    opened by ThatXliner 0
  • Update README for better comparison

    Update README for better comparison

    1. html2text is fast but not very configurable (there's only so any options)
    2. Tomd sucks
    3. Add an unmarker (with html2text-style configuration) to prove that unmarkd's configurability is at least equal to html2text
    documentation 
    opened by ThatXliner 0
  • Use a more reliable markdown parser

    Use a more reliable markdown parser

    Instead of using commonmark, maybe https://github.com/executablebooks/markdown-it-py, https://github.com/trentm/python-markdown2, https://github.com/lepture/mistune, or https://github.com/Python-Markdown/markdown.

    Also, I found tomd which might render this project useless 😬

    tests 
    opened by ThatXliner 0
  • Cannot handle nested bold and italics

    Cannot handle nested bold and italics

    When encountering input like <em>Italic and <strong>bold and italic</strong></em>, the output is wrong, usually shadowed by the outer tag (in this case, <em>)

    bug 
    opened by ThatXliner 0
  • Optimize code

    Optimize code

    I've noticed that unmarkers.BaseUnmarker been documented as an "abstract base class" when we're actually using it otherwise.

    Also, there's some dead code and we should actually sprinkle @staticmethod on some of them.

    Here's my idea:

    • Move all the tag_* methods in BaseUnmarker ➡️ BasicUnmarker
    • Rename: BaseUnmarker ➡️ AbstractUnmarker
    • Alias: BaseUnmarker ➡️ BasicMarker
    • Run shed on the whole codebase (with --refactor)

    Version bump: minor

    enhancement 
    opened by ThatXliner 0
  • Save CSS information

    Save CSS information

    1. Parse any css files or style tags found. Save it
    2. When a class attribute is found, try to resolve it to the css
    3. Add the resolved to the style attribute: convert to inline css
    enhancement 
    opened by ThatXliner 1
Releases(v0.1.9)
Owner
ThatXliner
I code Python. To me, programming is a logic puzzle. A fun one :D
ThatXliner
python's memory-saving dictionary data structure

ConstDict python代替的Dict数据结构 若字典不会增加字段,只读/原字段修改 使用ConstDict可节省内存 Dict()内存主要消耗的地方: 1、Dict扩容机制,预留内存空间 2、Dict也是一个对象,内部会动态维护__dict__,增加slot类属性可以节省内容 节省内存大小

Grenter 1 Nov 03, 2021
MySQL Connectivity based project. Contains various functions of a Store-Management-System

An Intermediate Level Python - MySQL Connectivity based project. Contains various functions of a Store-Management-System.

Yash Wadhvani 2 Nov 21, 2022
Covid-ml-predictors - COVID predictions using AI.

COVID Predictions This repo contains ML models to be trained on COVID-19 data from the UK, sourced off of Kaggle here. This uses many different ML mod

1 Jan 09, 2022
Percolation simulation using python

PythonPercolation Percolation simulation using python Exemple de percolation : Etude statistique sur le pourcentage de remplissage jusqu'à percolation

Tony Chouteau 1 Sep 08, 2022
Run Windows Applications on Linux as if they are native, Use linux applications to launch files files located in windows vm without needing to install applications on vm. With easy to use configuration GUI

Run Windows Applications on Linux as if they are native, Use linux applications to launch files files located in windows vm without needing to install applications on vm. With easy to use configurati

Casu Al Snek 2k Jan 02, 2023
Python Excuse Generator

Excuse Generator Python Excuse Generator This project is an excuse generator that provides the user with an excuse as to why they weren't paying atten

Collin Sanders 5 Jul 07, 2022
Python library for parsing Godot scene files

Godot Parser This is a python library for parsing Godot scene (.tscn) and resource (.tres) files. It's intended to make it easier to automate certain

Steven Arcangeli 30 Jan 04, 2023
List of short Codeforces problems with a statement of 1000 characters or less. Python script and data files included.

Shortest problems on Codeforces List of Codeforces problems with a short problem statement of 1000 characters or less. Sorted for each rating level. B

32 Dec 24, 2022
Wrapper for the undocumented CodinGame API. Can be used both synchronously and asynchronlously.

codingame API wrapper Pythonic wrapper for the undocumented CodinGame API. Installation Python 3.6 or higher is required. Install codingame with pip:

Takos 19 Jun 20, 2022
HomeAssistant Linux Companion

Application to run on linux desktop computer to provide sensors data to homeasssistant, and get notifications as if it was a mobile device.

Javier Lopez 10 Dec 27, 2022
CoreSE - basic of social Engineering tool

Core Social Engineering basic of social Engineering tool. just for fun :) About First of all, I must say that I wrote such a project because of my int

Hamed Mohammadvand 7 Jun 10, 2022
Find out where all films you want to watch are streaming

Just Watch Letterboxd Find out where all films you want to watch are streaming Ever wonder what films you want to watch are already on the streaming p

Jordan Oslislo 2 Feb 04, 2022
Driving lessons made simpler. Custom scheduling API built with Python.

NOTE This is a mirror of a GitLab repository. Dryvo Dryvo is a unique solution for the driving lessons industry. Our aim is to save the teacher’s time

Adam Goldschmidt 595 Dec 05, 2022
SQL centered, docker process running game

REQUIREMENTS Linux Docker Python/bash set up image "docker build -t game ." create db container "run my_whatever/game_docker/pdb create" # creating po

1 Jan 11, 2022
Python library for generating CycloneDX SBOMs

Python Library for generating CycloneDX This CycloneDX module for Python can generate valid CycloneDX bill-of-material document containing an aggregat

CycloneDX SBOM Standard 31 Dec 16, 2022
🤖️ Plugin for Sentry which allows sending notification via DingTalk robot.

Sentry DingTalk Sentry 集成钉钉机器人通知 Requirments sentry = 21.5.1 特性 发送异常通知到钉钉 支持钉钉机器人webhook设置关键字 配置环境变量 DINGTALK_WEBHOOK: Optional(string) DINGTALK_CUST

1 Nov 04, 2021
A reproduction repo for a Scheduling bug in AirFlow 2.2.3

A reproduction repo for a Scheduling bug in AirFlow 2.2.3

Ilya Strelnikov 1 Feb 09, 2022
This is a Saleae Logic custom high level analyzer that allows you to search and mark specific packets.

SaleaePacketParser This is a Saleae Logic custom high level analyzer that allows you to search and mark specific packets. Field "Search For" is used f

1 Dec 16, 2021
Pokehandy - Data web app sobre Pokémon TCG que desarrollo durante transmisiones de Twitch, 2022

⚡️ Pokéhandy – Pokémon Hand Simulator [WIP 🚧 ] This application aims to simulat

Rodolfo Ferro 5 Feb 23, 2022
Some basic sorting algos

Sorting-Algos Some basic sorting algos HacktoberFest 2021 This repository consists of mezzo-level projects that undertake a simple task and perform it

Manthan Ghasadiya 7 Dec 13, 2022