API to parse tibia.com content into python objects.

Last update: Oct 31, 2022

Overview

Tibia.py

An API to parse Tibia.com content into object oriented data.

No fetching is done by this module, you must provide the html content.

Features:

Converts data into well-structured Python objects.
Type consistent attributes.
All objects can be converted to JSON strings.
Can be used with any networking library.
Support for characters, guilds, houses and worlds, tournaments, forums, etc.

Installing

Install and update using pip

pip install tibia.py

Installing the latest version form GitHub

pip install git+https://github.com/Galarzaa90/tibia.py.git -U

Usage

This library is composed of two parts, parsers and an asynchronous request client.

The asynchronous client (tibiapy.Client) contains methods to obtain information from Tibia.com.

The parsing methods allow you to get Python objects given the html content of a page.

import tibiapy

# Asynchronously
import aiohttp

async def get_character(name):
    url = tibiapy.Character.get_url(name)

    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            content = await resp.text()
    character = tibiapy.Character.from_content(content)
    return character

# Synchronously
import requests

def get_character_sync(name):
    url = tibiapy.Character.get_url(name)
    
    r = requests.get(url)
    content = r.text
    character = tibiapy.Character.from_content(content)
    return character

Documentation

https://tibiapy.readthedocs.io/

Comments

DisplayItems attribute name is returning None
I was querying the items of a character at the bazaar and the DisplayItem attribute name is returning None

Here is my code snippet:

response : TibiaResponse = self.client.loop.run_until_complete(self.client.fetch_auction(character_auction_id, fetch_items=True)) details : AuctionDetails = response.data items : List[DisplayItem] = details.items.entries print(items)

The print of the returned list:

[<DisplayItem name=None count=22 item_id=238>, <DisplayItem name=None count=347 item_id=239>, <DisplayItem name=None count=11 item_id=266>]

Yesterday everything was working fine, the problem emerged this morning. Maybe the Tibia Bazaar website changed something in their HTML indexing?
opened by guriosam 6
Problem with the comma when there are more than 1000 players online.

Hello! Cipsoft added a comma in the world list when there are more than 1000 players online.

File "\site-packages\tibiapy\world.py", line 267, in from_content world._parse_world_info(tables.get("World Information", [])) File "\site-packages\tibiapy\world.py", line 344, in _parse_world_info self.online_count = int(world_info.pop("players_online")) ValueError: invalid literal for int() with base 10: '1,024'

opened by Pixelao 2
tibiapy.HousesSection.from_content is broken

Problem Method tibiapy.HousesSection.from_content(link to code) throws tibiapy.errors.InvalidContent error instead of returning a HousesSection object.

The reason this unintended behavior happens is because we try to access a Nonetype object inside HouseSection._parse_filters method when we call it house_results._parse_filters(tables["House Search"]) in line 172.

Quick fix Commenting line 172 solves the issue for me (at least it does for my use case).

opened by aaronga97 1
v2.4.0
Changes

Added support for multiple houses per character. Accessible on Character.houses field.

Character.house is now deprecated. It will contain the character's first house or None.
opened by Galarzaa90 1
v1.1.0
Changes

Parsing Highscores from Tibia.com and TibiaData.

Some strings from TibiaData had unpredictable trailing whitespaces, all leading and trailing whitespaces are removed.

Added type hints to many variables and methods.
opened by Galarzaa90 1
Parse character position

Character's position is not being parsed, for example, Steve has CipSoft Member position. This seems to be the only available position.

Not to be confused with the position field in Account Information.

Also, these characters show Cipsoft Member in their status when listed as "Other Characters".
enhancement

opened by Galarzaa90 1
Add HouseStatus and Sex enums

Testing the waters to maybe improve code. Those do not seem to be used in many places, so they are both good testing subjects.

Tests kept successful so it seems to be ok. This might have some limitations with json parsing constructors, so let's discuss what you think later :)

opened by Tschis 1
Refactor guild.py and unit tests
Refactor guild.py to split different sections of the parsing method. This makes it easier to test and analyze/maintain the code in the future;

Add new resource and unit tests for best case scenario on each section;
opened by Tschis 1
Parse character titles and main character information

Summer update 2019 introduced character titles, as well as a small icon that indicates which is the main character of the account in the other characters section.

opened by Galarzaa90 0
v1.0.0
Changes

Added support for TibiaData JSON parsing. To have interoperability between Tibia.com and TibiaData.

Added support for parsing Houses, House lists, World and World list

Added support for many missing attributes in Character and Guilds.

All objects are now serializable to JSON strings.
opened by Galarzaa90 0
Add /tests as a package

As seen here, this might help running the tests from the root folder.

Also added some changes that might facilitate changes into the current test case or new test cases to be added in the future (relative resources path).

opened by Tschis 0

Releases(v5.5.2)

v5.5.2(Sep 3, 2022)
Changes

Fixed Houses section not parsing due to a change in the filters table.

Fixed status parameter not generating the correct URL in the houses section.

Source code(tar.gz)
Source code(zip)
v5.5.1(Aug 2, 2022)
Changes

Adjusted parsing to support the changes related to mobile devices introduced on the day of this release. The following sections were affected:

Highscores

News

Forums

Spells

Source code(tar.gz)
Source code(zip)
v5.5.0(Jul 27, 2022)
Changes

Added BOSS_POINTS to Category in highscores.

Source code(tar.gz)
Source code(zip)
v5.4.0(Jul 23, 2022)
Changes

Added boss_points and bosstiary_progress to auctions.

Source code(tar.gz)
Source code(zip)
v5.3.0(Jul 23, 2022)
Changes

Added support for Boostable Bosses.

Source code(tar.gz)
Source code(zip)
v5.2.1(Mar 1, 2022)
Changes

Fixed bug in news ticker with a div tag failing to parse.

Updated code to detect the discussion thread link.

Source code(tar.gz)
Source code(zip)
v5.2.0(Dec 31, 2021)
Changes

Added exalted_dust and exalted_dust_limit attributes to auctions.

Source code(tar.gz)
Source code(zip)
v5.1.0(Sep 16, 2021)
Changes

Added traded attribute to death killers, to indicate that the killer was traded after the death occurred.

Properly handle deaths caused by summons of traded characters.

Source code(tar.gz)
Source code(zip)
v5.0.1.post0(Aug 26, 2021)

Source code(tar.gz)
Source code(zip)
v5.0.1(Aug 26, 2021)
Changes

Fixed many sections not being parsed correctly due to changes to Tibia.com.

Houses list

News list

Spells section

Forums section

Fixed character's houses failing to parse due to a bug in the display in Tibia.com.

Temporarily disabling this attribute.

Source code(tar.gz)
Source code(zip)
v5.0.1a1(Aug 24, 2021)
Prelease to address part of the Tibia.com changes.

Changes

Fixed character's houses failing to parse due to a bug in the display in Tibia.com.

Temporarily disabling this attribute.

Fixed house list not parsing correctly due to Tibia.com changes

Fixed news list not parsing correctly due to Tibia.com changes

Pending

Forums section

Spells section

Source code(tar.gz)
Source code(zip)
v5.0.0(Aug 7, 2021)
Changes

Added parsing for Tibia Drome leaderboards, new Leaderboard class.

Auxiliary classes LeaderboardEntry and LeaderboardRotation were added as well.

New Client method: fetch_leaderboards.

Added parsing for Spells library, new SpellsSection class.

Auxiliary classes Spell and SpellEntry.

New Client methods: fetch_spell and fetch_spells

Fix last page of highscores having 0 as page value.

Using the Client class, you can now fetch and parse content from the test version of www.tibia.com when available.

Note that if the test website has changes, parsing might not be possible.

Internal URL attributes might still point to the regular website.

New HousesSection class, including the house filtering attributes.

New NewsArchive class, including the news filtering attributes.

New GuildsSection class, to replace the lists of GuildEntry

Many "ListedObject" classes were renamed to "ObjectEntry", for details check the breaking changes below.

Fixed bug with other characters not being parsed.

Added traded attribute to LastPost class.

Added thread_starter_traded attribute to ThreadEntry

Breaking Changes:

Python 3.7 or higher is now required.

House classes no longer have get_list_url and list_from_content methods.

Client.fetch_world_houses now returns a HousesSection instance in its data attribute, instead of a list of ListedHouses.

ListedHouse.highest_bid attribute now may be None if the house's auction has not yet started.

ListedHouse class renamed to HouseEntry.

Removed deprecated property AuctionFilters.item.

Client.fetch_news_archive, Client.fetch_recent_news now returns an instance of NewsArchive in the data attribute.

ListedNews class renamed to NewsEntry.

News and NewsEntry no longer have a get_list_url method.

ListedBoard class renamed to BoardEntry.

ListedThread class renamed to ThreadEntry.

ListedAnnouncement class renamed to AnnouncementEntry.

ListedWorld class renamed to WorldEntry.

ListedAuction class renamed to AuctionEntry.

AuctionDetails class renamed to Auction.

ListedGuild class renamed to GuildEntry.

ListedTournament class renamed to TournamentEntry.

Creature class renamed to CreatureEntry.

CreatureDetail class renamed to Creature.

Guild and GuildEntry class no longer have a get_list_url method.

Renamed begin_date parameter to start_date in fetch_news_archive.

Renamed race attribute of CreatureEntry and Creature to identifier, method parameters renamed as well.

CreaturesSection.from_boosted_creature_header renamed to CreaturesSection.boosted_creature_from_header.

Source code(tar.gz)
Source code(zip)
v4.1.7(Jun 30, 2021)
Changes

Fixed titles being parsed as part of the name for guild members with symbols in their name.

Source code(tar.gz)
Source code(zip)
v4.1.6(Jun 28, 2021)
Changes

Fixed worlds not being parsed correctly again due to tournament worlds order changing. After this fix, the order should not matter anymore.

Source code(tar.gz)
Source code(zip)
v4.1.5(Jun 26, 2021)
Changes

Fixed parsing bug on characters, returning an incorrect exception when a character doesn't exist.

Source code(tar.gz)
Source code(zip)
v4.1.4(Jun 17, 2021)
Changes

Fixed worlds not being parsed correctly due to tournament worlds order changing.

Source code(tar.gz)
Source code(zip)
v4.1.3(May 12, 2021)
Changes

Fixed house transfer date not being parsed properly.

Source code(tar.gz)
Source code(zip)
v4.1.2(Apr 27, 2021)
Changes

Fixed parsing errors for forum posts that contained a copy of the signature separator in the signature.

Source code(tar.gz)
Source code(zip)
v4.1.1(Apr 19, 2021)
Changes

Fixed bug with extraneous character in some item descriptions, causing auction to give a parsing error.

Source code(tar.gz)
Source code(zip)
v4.1.0(Mar 30, 2021)
Changes

Added prey_wildcards attribute to AuctionDetails.

Added filters parameter to CharacterBazaar.get_auctions_history_url and Client.fetch_auction_history.

Source code(tar.gz)
Source code(zip)
v4.0.0(Mar 11, 2021)
Changes

Breaking change: Removed BoostedCreature class, replaced by Creature class.

Attributes should be compatible, image_url is a property of Creature, calculated from its race attribute.

Added parsing and fetching for the Creature library section.

Added CreatureSection, Creature, and CreatureDetail classes.

Added traded attribute to ForumAuthor. Indicates if the author was a traded character.

Previously, it would mark the author as a deleted character and its name would include (traded).

Fixed a bug with ForumBoards not parsing due to the cookies dialog that was added.

Added battleye_type attribute to ListedWorld and World classes. Indicates the type of BattlEye protection the world has.

battleye_protected is now a property instead of an attribute.

Added YELLOW and GREEN aliases to all BattlEye related enums.

Fixed wrong timezone being used for forum related dates.

Source code(tar.gz)
Source code(zip)
v3.7.1(Feb 16, 2021)
Changes

Adjusted highscores parsing for upcoming Tibia.com changes.

Source code(tar.gz)
Source code(zip)
v.3.7.0(Feb 9, 2021)
Changes

Parse familiars from auctions

Updated the way tooltips in auctions are parsed, the format changed, resulting in the previous code not working anymore.

Results count in bazaar pages are now properly parsed when there are comma thousand separators.

Item amounts are now more accurate instead of being based from their indicator (which was grouping them in thousands)

Source code(tar.gz)
Source code(zip)
v3.6.5(Jan 28, 2021)
Changes

Fixed auction history parsing breaking due to the cookie consent dialog.

Source code(tar.gz)
Source code(zip)
v3.6.4(Jan 26, 2021)
Changes

Fixed world list parsing breaking due to the cookie consent dialog.

Source code(tar.gz)
Source code(zip)
v3.6.3(Jan 15, 2021)
Changes

Fixed bug in guild names being parsed with Non-Breaking spaces instead of a regular space.

Source code(tar.gz)
Source code(zip)
v3.6.2(Jan 2, 2021)
Changes

Fixed bug in Event Calendar parsing.

Source code(tar.gz)
Source code(zip)
v3.6.1(Dec 28, 2020)
Changes

Fixed guild information being parsed incorrectly for characters in guilds containing "of the" in their name.

Source code(tar.gz)
Source code(zip)
v3.6.0(Dec 12, 2020)
Changes

Added support for the new filtering options in Highscores

Added battleye_filter and pvp_types_filter attributes.

Added get_page_url() instance method to Highscores class.

Added previous_page_url and next_page_url properties.

Source code(tar.gz)
Source code(zip)
v3.5.7(Dec 4, 2020)
Changes

Fixed bug in Event Calendar parsing.

Source code(tar.gz)
Source code(zip)

Owner

Allan Galarza

Mechatronic Engineer

GitHub Repository https://tibiapy.readthedocs.io/

Find thumbnails and original images from URL or HTML file.

Haul Find thumbnails and original images from URL or HTML file. Demo Hauler on Heroku Installation on Ubuntu $ sudo apt-get install build-essential py

150 Oct 15, 2022

Python script who crawl first shodan page and check DBLTEK vulnerability

🐛 MASS DBLTEK EXPLOIT CHECKER USING SHODAN 🕸 Python script who crawl first shodan page and check DBLTEK vulnerability

4 Jan 09, 2022

A high-level distributed crawling framework.

Cola: high-level distributed crawling framework Overview Cola is a high-level distributed crawling framework, used to crawl pages and extract structur

1.5k Jan 04, 2023

A simple proxy scraper that utilizes the requests module in python.

Proxy Scraper A simple proxy scraper that utilizes the requests module in python. Usage Depending on your python installation your commands may vary.

3 Sep 08, 2021

👁️ Tool for Data Extraction and Web Requests.

httpmapper 👁️ Project • Technologies • Installation • How it works • License Project 🚧 For educational purposes. This is a project that I developed,

15 Dec 05, 2021

Demonstration on how to use async python to control multiple playwright browsers for web-scraping

Playwright Browser Pool This example illustrates how it's possible to use a pool of browsers to retrieve page urls in a single asynchronous process. i

8 Oct 27, 2022

优化版本的京东茅台抢购神器

1.8k Mar 18, 2022

PS5 bot to find a console in france for chrismas 🎄🎅🏻 NOT FOR SCALPERS

Une PS5 pour Noël Python + Chrome --headless = une PS5 pour noël MacOS Installer chrome Tweaker le .yaml pour la listes sites a scrap et les criteres

3 Feb 13, 2022

Script for scrape user data like "id,username,fullname,followers,tweets .. etc" by Twitter's search engine .

TwitterScraper Script for scrape user data like "id,username,fullname,followers,tweets .. etc" by Twitter's search engine . Screenshot Data Users Only

19 Nov 17, 2022

Web scraped S&P 500 Data from Wikipedia using Pandas and performed Exploratory Data Analysis on the data.

Web scraped S&P 500 Data from Wikipedia using Pandas and performed Exploratory Data Analysis on the data. Then used Yahoo Finance to get the related stock data and displayed them in the form of chart

3 Sep 09, 2022

Scrapy-based cyber security news finder

Cyber-Security-News-Scraper Scrapy-based cyber security news finder Goal To keep up to date on the constant barrage of information within the field of

2 Nov 01, 2021

This project was created using Python technology and flask tools to scrape a music site

python-scrapping This project was created using Python technology and flask tools to scrape a music site You need to install the following packages to

1 Dec 07, 2021

script to scrape direct download links (ddls) from google drive index.

bhadoo Google Personal/Shared Drive Index scraper. A small script to scrape direct download links (ddls) of downloadable files from bhadoo google driv

53 Dec 16, 2022

A Web Scraper built with beautiful soup, that fetches udemy course information. Get udemy course information and convert it to json, csv or xml file

Udemy Scraper A Web Scraper built with beautiful soup, that fetches udemy course information. Installation Virtual Environment Firstly, it is recommen

15 May 17, 2022

Web Scraping Framework

Grab Framework Documentation Installation $ pip install -U grab See details about installing Grab on different platforms here http://docs.grablib.

2.3k Jan 04, 2023

Simple python tool for the purpose of swapping latinic letters with cirilic ones and vice versa in txt, docx and pdf files in Serbian language

Alpha Swap English This is a simple python tool for the purpose of swapping latinic letters with cirylic ones and vice versa, in txt, docx and pdf fil

3 May 31, 2022

API to parse tibia.com content into python objects.

Related tags

Overview

Tibia.py

Installing

Usage

Documentation

Comments

Changes

Changes

Changes

Releases(v5.5.2)

v5.5.2(Sep 3, 2022)

Changes

v5.5.1(Aug 2, 2022)

Changes

v5.5.0(Jul 27, 2022)

Changes

v5.4.0(Jul 23, 2022)

Changes

v5.3.0(Jul 23, 2022)

Changes

v5.2.1(Mar 1, 2022)

Changes

v5.2.0(Dec 31, 2021)

Changes

v5.1.0(Sep 16, 2021)

Changes

v5.0.1.post0(Aug 26, 2021)

v5.0.1(Aug 26, 2021)

Changes

v5.0.1a1(Aug 24, 2021)

Changes

Pending

v5.0.0(Aug 7, 2021)

Changes

v4.1.7(Jun 30, 2021)

Changes

v4.1.6(Jun 28, 2021)

Changes

v4.1.5(Jun 26, 2021)

Changes

v4.1.4(Jun 17, 2021)

Changes

v4.1.3(May 12, 2021)

Changes

v4.1.2(Apr 27, 2021)

Changes

v4.1.1(Apr 19, 2021)

Changes

v4.1.0(Mar 30, 2021)

Changes

v4.0.0(Mar 11, 2021)

Changes

v3.7.1(Feb 16, 2021)

Changes

v.3.7.0(Feb 9, 2021)

Changes

v3.6.5(Jan 28, 2021)

Changes

v3.6.4(Jan 26, 2021)

Changes

v3.6.3(Jan 15, 2021)

Changes

v3.6.2(Jan 2, 2021)

Changes

v3.6.1(Dec 28, 2020)

Changes

v3.6.0(Dec 12, 2020)

Changes

v3.5.7(Dec 4, 2020)

Changes

Owner

Allan Galarza

Find thumbnails and original images from URL or HTML file.

Python script who crawl first shodan page and check DBLTEK vulnerability

A high-level distributed crawling framework.

A simple proxy scraper that utilizes the requests module in python.

👁️ Tool for Data Extraction and Web Requests.

Demonstration on how to use async python to control multiple playwright browsers for web-scraping