API to parse tibia.com content into python objects.

Overview

Tibia.py

An API to parse Tibia.com content into object oriented data.

No fetching is done by this module, you must provide the html content.

GitHub Workflow Status codecov PyPI PyPI - Python Version PyPI - License

Features:

  • Converts data into well-structured Python objects.
  • Type consistent attributes.
  • All objects can be converted to JSON strings.
  • Can be used with any networking library.
  • Support for characters, guilds, houses and worlds, tournaments, forums, etc.

Installing

Install and update using pip

pip install tibia.py

Installing the latest version form GitHub

pip install git+https://github.com/Galarzaa90/tibia.py.git -U

Usage

This library is composed of two parts, parsers and an asynchronous request client.

The asynchronous client (tibiapy.Client) contains methods to obtain information from Tibia.com.

The parsing methods allow you to get Python objects given the html content of a page.

import tibiapy

# Asynchronously
import aiohttp

async def get_character(name):
    url = tibiapy.Character.get_url(name)

    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            content = await resp.text()
    character = tibiapy.Character.from_content(content)
    return character

# Synchronously
import requests

def get_character_sync(name):
    url = tibiapy.Character.get_url(name)
    
    r = requests.get(url)
    content = r.text
    character = tibiapy.Character.from_content(content)
    return character

Documentation

https://tibiapy.readthedocs.io/

Comments
  • DisplayItems attribute name is returning None

    DisplayItems attribute name is returning None

    I was querying the items of a character at the bazaar and the DisplayItem attribute name is returning None

    Here is my code snippet:

    response : TibiaResponse = self.client.loop.run_until_complete(self.client.fetch_auction(character_auction_id, 
            fetch_items=True))
            
    details : AuctionDetails = response.data
    
    items : List[DisplayItem] = details.items.entries
    
    print(items)
    

    The print of the returned list:

    [<DisplayItem name=None count=22 item_id=238>, <DisplayItem name=None count=347 item_id=239>, <DisplayItem name=None count=11 item_id=266>]

    Yesterday everything was working fine, the problem emerged this morning. Maybe the Tibia Bazaar website changed something in their HTML indexing?

    opened by guriosam 6
  • Problem with the comma when there are more than 1000 players online.

    Problem with the comma when there are more than 1000 players online.

    Hello! Cipsoft added a comma in the world list when there are more than 1000 players online.

    File "\site-packages\tibiapy\world.py", line 267, in from_content world._parse_world_info(tables.get("World Information", [])) File "\site-packages\tibiapy\world.py", line 344, in _parse_world_info self.online_count = int(world_info.pop("players_online")) ValueError: invalid literal for int() with base 10: '1,024'

    opened by Pixelao 2
  • tibiapy.HousesSection.from_content is broken

    tibiapy.HousesSection.from_content is broken

    Problem Method tibiapy.HousesSection.from_content(link to code) throws tibiapy.errors.InvalidContent error instead of returning a HousesSection object.

    The reason this unintended behavior happens is because we try to access a Nonetype object inside HouseSection._parse_filters method when we call it house_results._parse_filters(tables["House Search"]) in line 172.

    Quick fix Commenting line 172 solves the issue for me (at least it does for my use case).

    opened by aaronga97 1
  • v2.4.0

    v2.4.0

    Changes

    • Added support for multiple houses per character. Accessible on Character.houses field.
    • Character.house is now deprecated. It will contain the character's first house or None.
    opened by Galarzaa90 1
  • v1.1.0

    v1.1.0

    Changes

    • Parsing Highscores from Tibia.com and TibiaData.
    • Some strings from TibiaData had unpredictable trailing whitespaces, all leading and trailing whitespaces are removed.
    • Added type hints to many variables and methods.
    opened by Galarzaa90 1
  • Parse character position

    Parse character position

    Character's position is not being parsed, for example, Steve has CipSoft Member position. This seems to be the only available position.

    Not to be confused with the position field in Account Information.

    Also, these characters show Cipsoft Member in their status when listed as "Other Characters".

    enhancement 
    opened by Galarzaa90 1
  • Add HouseStatus and Sex enums

    Add HouseStatus and Sex enums

    Testing the waters to maybe improve code. Those do not seem to be used in many places, so they are both good testing subjects.

    Tests kept successful so it seems to be ok. This might have some limitations with json parsing constructors, so let's discuss what you think later :)

    opened by Tschis 1
  • Refactor guild.py and unit tests

    Refactor guild.py and unit tests

    • Refactor guild.py to split different sections of the parsing method. This makes it easier to test and analyze/maintain the code in the future;
    • Add new resource and unit tests for best case scenario on each section;
    opened by Tschis 1
  • Parse character titles and main character information

    Parse character titles and main character information

    Summer update 2019 introduced character titles, as well as a small icon that indicates which is the main character of the account in the other characters section.

    opened by Galarzaa90 0
  • v1.0.0

    v1.0.0

    Changes

    • Added support for TibiaData JSON parsing. To have interoperability between Tibia.com and TibiaData.
    • Added support for parsing Houses, House lists, World and World list
    • Added support for many missing attributes in Character and Guilds.
    • All objects are now serializable to JSON strings.
    opened by Galarzaa90 0
  • Add /tests as a package

    Add /tests as a package

    As seen here, this might help running the tests from the root folder.

    Also added some changes that might facilitate changes into the current test case or new test cases to be added in the future (relative resources path).

    opened by Tschis 0
Releases(v5.5.2)
  • v5.5.2(Sep 3, 2022)

    Changes

    • Fixed Houses section not parsing due to a change in the filters table.
    • Fixed status parameter not generating the correct URL in the houses section.
    Source code(tar.gz)
    Source code(zip)
  • v5.5.1(Aug 2, 2022)

    Changes

    • Adjusted parsing to support the changes related to mobile devices introduced on the day of this release. The following sections were affected:
      • Highscores
      • News
      • Forums
      • Spells
    Source code(tar.gz)
    Source code(zip)
  • v5.5.0(Jul 27, 2022)

  • v5.4.0(Jul 23, 2022)

  • v5.3.0(Jul 23, 2022)

  • v5.2.1(Mar 1, 2022)

  • v5.2.0(Dec 31, 2021)

  • v5.1.0(Sep 16, 2021)

    Changes

    • Added traded attribute to death killers, to indicate that the killer was traded after the death occurred.
    • Properly handle deaths caused by summons of traded characters.
    Source code(tar.gz)
    Source code(zip)
  • v5.0.1.post0(Aug 26, 2021)

  • v5.0.1(Aug 26, 2021)

    Changes

    • Fixed many sections not being parsed correctly due to changes to Tibia.com.
      • Houses list
      • News list
      • Spells section
      • Forums section
    • Fixed character's houses failing to parse due to a bug in the display in Tibia.com.
      • Temporarily disabling this attribute.
    Source code(tar.gz)
    Source code(zip)
  • v5.0.1a1(Aug 24, 2021)

    Prelease to address part of the Tibia.com changes.

    Changes

    • Fixed character's houses failing to parse due to a bug in the display in Tibia.com.
      • Temporarily disabling this attribute.
    • Fixed house list not parsing correctly due to Tibia.com changes
    • Fixed news list not parsing correctly due to Tibia.com changes

    Pending

    • Forums section
    • Spells section
    Source code(tar.gz)
    Source code(zip)
  • v5.0.0(Aug 7, 2021)

    Changes

    • Added parsing for Tibia Drome leaderboards, new Leaderboard class.
      • Auxiliary classes LeaderboardEntry and LeaderboardRotation were added as well.
      • New Client method: fetch_leaderboards.
    • Added parsing for Spells library, new SpellsSection class.
      • Auxiliary classes Spell and SpellEntry.
      • New Client methods: fetch_spell and fetch_spells
    • Fix last page of highscores having 0 as page value.
    • Using the Client class, you can now fetch and parse content from the test version of www.tibia.com when available.
      • Note that if the test website has changes, parsing might not be possible.
      • Internal URL attributes might still point to the regular website.
    • New HousesSection class, including the house filtering attributes.
    • New NewsArchive class, including the news filtering attributes.
    • New GuildsSection class, to replace the lists of GuildEntry
    • Many "ListedObject" classes were renamed to "ObjectEntry", for details check the breaking changes below.
    • Fixed bug with other characters not being parsed.
    • Added traded attribute to LastPost class.
    • Added thread_starter_traded attribute to ThreadEntry

    Breaking Changes:

    • Python 3.7 or higher is now required.
    • House classes no longer have get_list_url and list_from_content methods.
    • Client.fetch_world_houses now returns a HousesSection instance in its data attribute, instead of a list of ListedHouses.
    • ListedHouse.highest_bid attribute now may be None if the house's auction has not yet started.
    • ListedHouse class renamed to HouseEntry.
    • Removed deprecated property AuctionFilters.item.
    • Client.fetch_news_archive, Client.fetch_recent_news now returns an instance of NewsArchive in the data attribute.
    • ListedNews class renamed to NewsEntry.
    • News and NewsEntry no longer have a get_list_url method.
    • ListedBoard class renamed to BoardEntry.
    • ListedThread class renamed to ThreadEntry.
    • ListedAnnouncement class renamed to AnnouncementEntry.
    • ListedWorld class renamed to WorldEntry.
    • ListedAuction class renamed to AuctionEntry.
    • AuctionDetails class renamed to Auction.
    • ListedGuild class renamed to GuildEntry.
    • ListedTournament class renamed to TournamentEntry.
    • Creature class renamed to CreatureEntry.
    • CreatureDetail class renamed to Creature.
    • Guild and GuildEntry class no longer have a get_list_url method.
    • Renamed begin_date parameter to start_date in fetch_news_archive.
    • Renamed race attribute of CreatureEntry and Creature to identifier, method parameters renamed as well.
    • CreaturesSection.from_boosted_creature_header renamed to CreaturesSection.boosted_creature_from_header.
    Source code(tar.gz)
    Source code(zip)
  • v4.1.7(Jun 30, 2021)

  • v4.1.6(Jun 28, 2021)

    Changes

    • Fixed worlds not being parsed correctly again due to tournament worlds order changing. After this fix, the order should not matter anymore.
    Source code(tar.gz)
    Source code(zip)
  • v4.1.5(Jun 26, 2021)

  • v4.1.4(Jun 17, 2021)

  • v4.1.3(May 12, 2021)

  • v4.1.2(Apr 27, 2021)

  • v4.1.1(Apr 19, 2021)

  • v4.1.0(Mar 30, 2021)

    Changes

    • Added prey_wildcards attribute to AuctionDetails.
    • Added filters parameter to CharacterBazaar.get_auctions_history_url and Client.fetch_auction_history.
    Source code(tar.gz)
    Source code(zip)
  • v4.0.0(Mar 11, 2021)

    Changes

    • Breaking change: Removed BoostedCreature class, replaced by Creature class.
      • Attributes should be compatible, image_url is a property of Creature, calculated from its race attribute.
    • Added parsing and fetching for the Creature library section.
      • Added CreatureSection, Creature, and CreatureDetail classes.
    • Added traded attribute to ForumAuthor. Indicates if the author was a traded character.
      • Previously, it would mark the author as a deleted character and its name would include (traded).
    • Fixed a bug with ForumBoards not parsing due to the cookies dialog that was added.
    • Added battleye_type attribute to ListedWorld and World classes. Indicates the type of BattlEye protection the world has.
      • battleye_protected is now a property instead of an attribute.
    • Added YELLOW and GREEN aliases to all BattlEye related enums.
    • Fixed wrong timezone being used for forum related dates.
    Source code(tar.gz)
    Source code(zip)
  • v3.7.1(Feb 16, 2021)

  • v.3.7.0(Feb 9, 2021)

    Changes

    • Parse familiars from auctions
    • Updated the way tooltips in auctions are parsed, the format changed, resulting in the previous code not working anymore.
    • Results count in bazaar pages are now properly parsed when there are comma thousand separators.
    • Item amounts are now more accurate instead of being based from their indicator (which was grouping them in thousands)
    Source code(tar.gz)
    Source code(zip)
  • v3.6.5(Jan 28, 2021)

  • v3.6.4(Jan 26, 2021)

  • v3.6.3(Jan 15, 2021)

  • v3.6.2(Jan 2, 2021)

  • v3.6.1(Dec 28, 2020)

  • v3.6.0(Dec 12, 2020)

    Changes

    • Added support for the new filtering options in Highscores
      • Added battleye_filter and pvp_types_filter attributes.
    • Added get_page_url() instance method to Highscores class.
    • Added previous_page_url and next_page_url properties.
    Source code(tar.gz)
    Source code(zip)
  • v3.5.7(Dec 4, 2020)

Owner
Allan Galarza
Mechatronic Engineer
Allan Galarza
Scraping script for stats on covid19 pandemic status in Chiba prefecture, Japan

About 千葉県の地域別の詳細感染者統計(Excelファイル) をCSVに変換し、かつ地域別の日時感染者集計値を出力するスクリプトです。 Requirement POSIX互換なシェル, e.g. GNU Bash (1) curl (1) python = 3.8 pandas = 1.1.

Conv4Japan 1 Nov 29, 2021
茅台抢购最新优化版本,茅台秒杀,优化了抢购协程队列

茅台抢购最新优化版本,茅台秒杀,优化了抢购协程队列

MaoTai 33 Sep 03, 2022
Quick Project made to help scrape Lexile and Atos(AR) levels from ISBN

Lexile-Atos-Scraper Quick Project made to help scrape Lexile and Atos(AR) levels from ISBN You will need to install the chrome webdriver if you have n

1 Feb 11, 2022
Collection of code files to scrap different kinds of websites.

STW-Collection Scrap The Web Collection; blog posts. This repo contains Scrapy sample code to scrap the following kind of websites: Do you want to lea

Tapasweni Pathak 15 Jun 08, 2022
Instagram profile scrapper with python

IG Profile Scrapper Instagram profile Scrapper Just type the username, and boo! :D Instalation clone this repo to your computer git clone https://gith

its Galih 6 Nov 07, 2022
Scraping weather data using Python to receive umbrella reminders

A Python package which scrapes weather data from google and sends umbrella reminders to specified email at specified time daily.

Edula Vinay Kumar Reddy 1 Aug 23, 2022
A simple reddit scraper to get memes (only images) from r/ProgrammerHumor.

memey A simple reddit scraper to get memes (only images) from r/ProgrammerHumor. Note Only works if you have firefox installed (yet). Instructions foo

2 Nov 16, 2021
a way to scrape a database of all of the isef projects

ISEF Database This is a simple web scraper which gets all of the projects and abstract information from here. My goal for this is for someone to get i

William Kaiser 1 Mar 18, 2022
Auto Join: A GitHub action script to automatically invite everyone to the organization who star your repository.

Auto Invite To The Organization By Star A GitHub Action script to automatically invite everyone to your organization that stars your repository. What

Max Base 11 Dec 11, 2022
A Spider for BiliBili comments with a simple API server.

BiliComment A spider for BiliBili comment. Spider Usage Put config.json into config directory, and then python . ./config/config.json. A example confi

Hao 3 Jul 05, 2021
Haphazard scripts for scraping bitcoin/bitcoin data from GitHub

This is a quick-and-dirty tool used to scrape bitcoin/bitcoin pull request and commentary data. Each output/pr number folder contains comments.json:

James O'Beirne 8 Oct 12, 2022
中国大学生在线 四史自动答题刷分(现仅支持英雄篇)

中国大学生在线 “四史”学习教育竞答 自动答题 刷分 (现仅支持英雄篇,已更新可用) 若对您有所帮助,记得点个Star 🌟 !!! 中国大学生在线 “四史”学习教育竞答 自动答题 刷分 (现仅支持英雄篇,已更新可用) 🥰 🥰 🥰 依赖 本项目依赖的第三方库: requests 在终端执行以下

XWhite 229 Dec 12, 2022
🥫 The simple, fast, and modern web scraping library

About gazpacho is a simple, fast, and modern web scraping library. The library is stable, actively maintained, and installed with zero dependencies. I

Max Humber 692 Dec 22, 2022
Python based Web Scraper which can discover javascript files and parse them for juicy information (API keys, IP's, Hidden Paths etc)

Python based Web Scraper which can discover javascript files and parse them for juicy information (API keys, IP's, Hidden Paths etc).

Amit 6 Aug 26, 2022
Nekopoi scraper using python3

Features Scrap from url Todo [+] Search by genre [+] Search by query [+] Scrap from homepage Example # Hentai Scraper from nekopoi import Hent

MhankBarBar 9 Apr 06, 2022
Web crawling framework based on asyncio.

Web crawling framework for everyone. Written with asyncio, uvloop and aiohttp. Requirements Python3.5+ Installation pip install gain pip install uvloo

Jiuli Gao 2k Jan 05, 2023
Web scraped S&P 500 Data from Wikipedia using Pandas and performed Exploratory Data Analysis on the data.

Web scraped S&P 500 Data from Wikipedia using Pandas and performed Exploratory Data Analysis on the data. Then used Yahoo Finance to get the related stock data and displayed them in the form of chart

Samrat Mitra 3 Sep 09, 2022
Generate a repository with mirror links for DriveDroid app

DriveDroid Repository Generator Generate a repository for the app that allow boot a PC using ISO files stored on your Android phone Check also an offi

Evgeny 11 Nov 19, 2022
A package that provides you Latest Cyber/Hacker News from website using Web-Scraping.

cybernews A package that provides you Latest Cyber/Hacker News from website using Web-Scraping. Latest Cyber/Hacker News Using Webscraping Developed b

Hitesh Rana 4 Jun 02, 2022
for those who dont want to pay $10/month for high school game footage with ads

nfhs-scraper Disclaimer: I am in no way responsible for what you choose to do with this script and guide. I do not endorse avoiding paywalls or any il

Conrad Crawford 5 Apr 12, 2022