Tool for installing and updating MiSTer cores and other files

Overview

MiSTer Downloader

This tool installs and updates all the cores and other extra files for your MiSTer. It also updates the menu core, the MiSTer firmware and the Linux system. The source for all downloads is the MiSTer Distribution repository.

The MiSTer Downloader is a substitute for the MiSTer Updater, and is meant to offer a more safe and robust experience, while being much faster.

As a drawback, the Downloader is not backwards compatible with the old INI files that were configured for the MiSTer Updater. In fact, as of today, this tool doesn't implement many fine-grained features that allow you to customize the updating process in depth. In case you value these features, consider to keep using the MiSTer Updater as usual. Both tools will coexist in the near future.

Setup and Usage

Download this ZIP file and extract downloader.sh to your /Scripts folder on your primary SD card (create that folder if it doesn't exist). You only need to perform this operation once, since this tool self-updates itself.

To use it, on your MiSTer main menu, go to the Scripts screen, and select downloader.

Options

You may create a /media/fat/downloader.ini file to tweak some parameters.

Here you can see the default parameters and the options that you may change:

[MiSTer]
; base_path is where most files will be installed
;   Useful for setups with USB storage, for example: '/media/usb0/'
base_path = '/media/fat/'

; base_system_path is where system files such as 'MiSTer' and 'menu.rbf' will be installed.
;   Warning: It is recommended to NOT change this setting regardless of your setup.
base_system_path = '/media/fat/'

; allow_delete options:
;   0 -> Don't allow this tool to delete anything at all.
;   1 -> Allow this tool to delete any old file from previous updates.
;   2 -> Allow this tool to delete only old cores that receive a new version.
allow_delete = 1

; allow_reboot options:
;   0 -> Don't allow this tool to ever reboot automatically.
;   1 -> Allow this tool to reboot the system after any system file has been updated.
;   2 -> Allow this tool to reboot the system only after Linux has been updated.
allow_reboot = 1

; update_linux options:
;   true -> Updates Linux when there is a new update (very recommended).
;   false -> Doesn't update Linux.
update_linux = true

; parallel_update options:
;   true -> Tries to more than one file simultaneously.
;   false -> Will only download one file at a time.
parallel_update = true

; downloader_timeout: Can be tweaked to increase the timeout time in seconds
;   It is useful to increase this value for users with slow connections.
downloader_timeout = 300

; downloader_retries: Can be tweaked to increase the retries per failed download
;   It is useful to increase this value for users with very unstable connections.
downloader_retries = 3

Roadmap

  • Initial Release
  • Cheats fetching
  • First-run optimisations
  • Configurable custom download filters
  • Handle duplicated games folders through symlinks (GBA <-> GBA2P, and GAMEBOY <-> GAMEBOY2P)
  • Integration with MiSTer binary
Comments
  • mra_alternatives don't seem to get updated

    mra_alternatives don't seem to get updated

    If the hash for the mra_alternatives summary file doesn't match the newest, mra_alternatives_summary.json.zip gets downloaded and the hashes are updated in the store (summary and files) but the files themselves aren't updated.

    opened by lagomorph 12
  • Unable to update

    Unable to update

    When I try to run aupdate from the MiSTer menu the part where it updates the CA-certs work. But after that it fails.

    The script will run normally now:
    Running MiSTer Downloader
    
    bash: line 122: BASH_SOURCE[0]: unbound variable
    Press any key to continue
    
    opened by Dankoman 6
  • error: invalid zip file with overlapped components (possible zip bomb)

    error: invalid zip file with overlapped components (possible zip bomb)

    I started receiving errors when trying to update today. Unzip is throwing an exception because it is recognizing the downloaded file as malicious. I am running the latest version of download.sh. The same issue occurs with any update script which downloads zipped files. Previously had no issues updating.

    /root# cat /media/fat/Scripts/.config/downloader/downloader.log START!

    Reading file: /media/fat/downloader.ini Reading 'distribution_mister' db section Reading 'jtcores' db section env: { "DOWNLOADER_LAUNCHER_PATH": "/media/fat/Scripts/downloader.sh", "DOWNLOADER_INI_PATH": null, "CURL_SSL": "--cacert /etc/ssl/certs/cacert.pem", "COMMIT": "856b6d5", "ALLOW_REBOOT": null, "UPDATE_LINUX": "true", "DEFAULT_DB_URL": "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip", "DEFAULT_DB_ID": "distribution_mister", "DEFAULT_BASE_PATH": null, "DEBUG": "false", "FAIL_ON_FILE_ERROR": "false" } config: { "databases": { "distribution_mister": { "db_url": "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip", "section": "distribution_mister" }, "jtcores": { "db_url": "https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip", "section": "jtcores" } }, "config_path": "/media/fat/downloader.ini", "base_path": "/media/fat", "base_system_path": "/media/fat", "storage_priority": "prefer_sd", "allow_delete": 1, "allow_reboot": 1, "update_linux": true, "parallel_update": true, "downloader_size_mb_limit": 100, "downloader_process_limit": 300, "downloader_timeout": 300, "downloader_retries": 3, "zip_file_count_threshold": 60, "zip_accumulated_mb_threshold": 100, "filter": null, "verbose": false, "debug": false, "default_db_id": "distribution_mister", "start_time": 1654702379.9457989, "user_defined_options": [], "curl_ssl": "--cacert /etc/ssl/certs/cacert.pem", "update_linux_environment": 0, "fail_on_file_error": false, "commit": "856b6d5" } cacert file at "/etc/ssl/certs/cacert.pem" seems to be fine. Loading db from url: https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip Loading db from url: https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip Downloading 2 files: /tmp/temp_filet7etzsji /tmp/temp_filex45tz6id *..

    Checking hashes... ++ subprocess.run unzip -p /tmp/temp_filet7etzsji Return Code was '12'

    error: invalid zip file with overlapped components (possible zip bomb)

    Could not load json from "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip" subprocess.run unzip -p /tmp/temp_filex45tz6id Return Code was '12'

    error: invalid zip file with overlapped components (possible zip bomb)

    Could not load json from "https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip" Skipping local_store saving...

    =========================== Downloader 1.5 (856b6d5) by theypsilon. Run time: 0:00:02.29s Log: /media/fat/Scripts/.config/downloader/downloader.log

    Installed: none.

    Errors: https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip, https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip

    linux_descriptions_count: 0 Length of failed_dbs: 2

    opened by LuoJohnT 5
  • Question: Save updates to /media folder without appending _Consoles/_Arcade/etc folders

    Question: Save updates to /media folder without appending _Consoles/_Arcade/etc folders

    Hi there,

    Is there any way to set the downloader to update and install the files I want straight into the /media folder instead of adding the associated _Console folder or any other similar folder? I don't use a ton of cores so I'd love to just run the script and keep everything clean. This way, I can just boot the MiSTer, and immediately see the cores in the list without entering additional directories.

    (Unless this is already possible)

    opened by Chromagram 3
  • downloader failed:

    downloader failed: "Could not load storage"

    Hello,

    I am attempting to run update_all.sh script with the downloader enabled. The latest jtbeta.zip file is in my 'mame' folder and all of the necessary options are enabled, as far as I can tell, in the update_all options menu (i.e. Install Premium Cores, Install Unofficial Cores etc.) I am recieving the following error:


    Reading file: /media/fat/download.ini Reading 'distribution_mister' db section Reading 'jtcores' db section Reading 'theypsilon_unofficial_distribution" db section Could not load storage Could not load json from "https://raw.githubusercontent.com/MiSTer--devel/Distribution_MiSTer/main/db.json.zip" Could not load json from "https://raw.githubusercontent.com/jotego/jtpremium/main/jtbindb.json.zip" Could not load json from "https://raw.githubusercontent.com/theypsilon/Distribution_Unofficial_MiSTer/unofficialdb.json.zip"

    There were some errors in the Updaters. Therefore, MiSTer hasn't been fully updated.

    Check these logs from the Updaters that failed:

    • /media/fat/Scripts/ .config/downloader/downloader1.log
    • BIOS-GETTER
    • /media/fat/Scripts/ .config/downloader/downloader2.log

    I receive the same error when running just the downloader.sh script. All official cores were downloaded and installed correctly using update_all.

    Thank you!

    opened by eebaser 3
  • Issue wit pc downloader

    Issue wit pc downloader

    Hi,

    I get the following error from https://github.com/MiSTer-devel/Downloader_MiSTer/releases/download/latest/downloader.zip

    {code} Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in run_code File "c:\TEMP\roka\2\tmp2bdrzqig.zip_main.py", line 49, in NameError: name 'exit' is not defined {code}

    sys.exit instead of exit is needed

    opened by RolandKaechele 2
  • Refactor downloader scripts

    Refactor downloader scripts

    While I was reading the scripts (to understand how they work) I've started putting some comments at some places. Later I've started introducing some "echo" statement as well. Also, I've noticed /etc/ssl/certs/cacert.pem is used in many places as a string literal, so I've created a constant to pass this around. In the end, I've tried to run this outside of MiSTer and I got hit by some missing binaries, so I added a function to check for those.

    Nothing significant here, mostly cosmetic changes I would say. Feel free to approve or reject this.

    opened by drpaneas 2
  • Error unpacking BIOS zips

    Error unpacking BIOS zips

    I'm on a fresh install and I am getting this error during the BiosDB step (in update_all):

    Extracting NeoGeo UniBios from http://unibios.free.fr
    checkdir:  cannot create extraction directory: /tmp/q48as6j6/neogeo_unibios
               No such file or directory
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 95, in download_dbs_contents
        db_importer.process_changed_files(changed_files, needed_zips, filtered_zip_data)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 586, in process_changed_files
        self._import_zip_contents(needed_zips, filtered_zip_data, file_downloader)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 672, in _import_zip_contents
        self._file_system.unzip_contents(temp_zip, tmp_path, list(zipped_files['files']))
      File "/tmp/dont_download.zip/downloader/file_system.py", line 335, in unzip_contents
        raise Exception("Could not unzip %s: %s" % (file, result.returncode))
    Exception: Could not unzip /tmp/hir0qr7h_neogeo_unibios_contents.zip: 2
    

    I think the parent tmp directory isn't being created after the name is generated?

    https://github.com/MiSTer-devel/Downloader_MiSTer/blob/0060984366a335546a7152ff83065c729d962d83/src/downloader/online_importer.py#L671

    opened by wizzomafizzo 2
  • Failure to remove directory when using retronas and MiSTer

    Failure to remove directory when using retronas and MiSTer

    I opened an issue for this over in the retronas repo, but it might be better suited here.

    Copy/paste from the original issue:


    Hi!

    Just getting started with retronas, and connected my MiSTer setup to it over CIFS. When I run downloader.sh, it tries to remove the empty folder for ATARI2600, but ends up failing:

    Deleting empty folder /media/fat/games/ATARI2600
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 108, in download_dbs_contents
        self._remove_folders(importer_command)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 210, in _remove_folders
        system_file_system.remove_folder(full_folder_path)
      File "/tmp/dont_download.zip/downloader/file_system.py", line 239, in remove_folder
        os.rmdir(self._path(path))
    NotADirectoryError: [Errno 20] Not a directory: '/media/fat/games/ATARI2600'
    

    This makes the rest of the update/download fail as well. Creating an empty file in the ATARI2600 folder removes this failure point.

    opened by jonasrosland 2
  • Won't read from /media/fat/downloader.ini

    Won't read from /media/fat/downloader.ini

    Readme says to use this ini to make changes, but it never reads from it. It's only looking at this one according to my log: Reading file: /media/fat/Update/MiSTer_Downloader/downloader.ini

    My ini: https://pastebin.com/DNPrxXEJ

    Log: https://pastebin.com/1HwS6gh8

    I set verbose to true and log says it's false Also tried moving the ini to "/media/fat/Update/MiSTer_Downloader/downloader.ini" and it still doesn't read it.

    opened by djvj1 2
  • Updates Not Downloading After Adding downloader.ini

    Updates Not Downloading After Adding downloader.ini

    So I created downloader.ini, copy/pasted the defaults and saved it into my root directory (where MiSTer.ini) lives. I noticed that when I run the downloader script now, nothing updates. After doing some research, I found that the downloader.ini needs to point to the default MiSTer [distribution_mister] database.

    If this is the case, then shouldn't this be documented and added to the list of default ini settings?

    opened by mjj03301977 2
  •  Invalid cross-device link on update

    Invalid cross-device link on update

    Copying new MiSTer binary:
    Traceback (most recent call last):
      File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
        exit_code = execute_full_run(
      File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
        exit_code = runner.full_run()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
        result = self._full_run_impl()
      File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
        self._online_importer.download_dbs_contents(importer_command, full_resync)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 95, in download_dbs_contents
        db_importer.process_changed_files(changed_files, needed_zips, filtered_zip_data)
      File "/tmp/dont_download.zip/downloader/online_importer.py", line 588, in process_changed_files
        file_downloader.download_files(self._is_first_run())
      File "/tmp/dont_download.zip/downloader/file_downloader.py", line 121, in download_files
        self._file_system.move(FILE_MiSTer_new, FILE_MiSTer)
      File "/tmp/dont_download.zip/downloader/file_system.py", line 189, in move
        os.replace(self._path(source), self._path(target))
    OSError: [Errno 18] Invalid cross-device link: '/media/usb0/MiSTer.new' -> '/media/fat/MiSTer'
    
    Downloader failed!
    

    Here is my download.ini

    [MiSTer]
    base_path = '/media/usb0'
    
    [distribution_mister]
    db_url = https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip
    
    [jtcores]
    db_url = https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip
    
    [bios_db]
    db_url = https://raw.githubusercontent.com/theypsilon/BiosDB_MiSTer/db/bios_db.json
    
    opened by KClough 2
  • Delete old config/DIP switch files when needed

    Delete old config/DIP switch files when needed

    A common source of problems is when the core logic changes leaving the saved config/DIP files unusable and possibly breaking some functionality if used with old files.

    For DIP switches, it is easy to detect a change by comparing the dip switch section inside the MRA with the previous MRA file. If a change in the DIP sw. section is detected, the downloader should delete the old file in the config/dips folder. This would apply to any MRA of any arcade core.

    For config file changes, I don't think it's easy to detect automatically because the configuration information shown in the OSD is embedded in the RBF file and has gone through some transformations. So it isn't plain ASCII. Maybe the downloader could read some metadata in the repositories. I could, for instance, make the config string available as plain text in the version.log file of each core. Currently, this file points to the source code git commit used to produce a specific RBF in JTBIN.

    Please consider implementing these improvements.

    opened by jotego 5
Code basis for the paper "Camera Condition Monitoring and Readjustment by means of Noise and Blur" (2021)

Camera Condition Monitoring and Readjustment by means of Noise and Blur This repository contains the source code of the paper: Wischow, M., Gallego, G

7 Dec 22, 2022
Vision-and-Language Navigation in Continuous Environments using Habitat

Vision-and-Language Navigation in Continuous Environments (VLN-CE) Project Website — VLN-CE Challenge — RxR-Habitat Challenge Official implementations

Jacob Krantz 132 Jan 02, 2023
Official Pytorch Implementation of 3DV2021 paper: SAFA: Structure Aware Face Animation.

SAFA: Structure Aware Face Animation (3DV2021) Official Pytorch Implementation of 3DV2021 paper: SAFA: Structure Aware Face Animation. Getting Started

QiulinW 122 Dec 23, 2022
Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

Patch-Rotation(PatchRot) Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models Submitted to Neurips2021 To

4 Jul 12, 2021
Code for the paper Learning the Predictability of the Future

Learning the Predictability of the Future Code from the paper Learning the Predictability of the Future. Website of the project in hyperfuture.cs.colu

Computer Vision Lab at Columbia University 139 Nov 18, 2022
NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering Paper: https://arxiv.org/abs/2103.00762 Running Run on the provided DTU scene cd run ba

Fanbo Xiang 67 Dec 28, 2022
Fuzzification helps developers protect the released, binary-only software from attackers who are capable of applying state-of-the-art fuzzing techniques

About Fuzzification Fuzzification helps developers protect the released, binary-only software from attackers who are capable of applying state-of-the-

gts3.org (<a href=[email protected])"> 55 Oct 25, 2022
This is a Python Module For Encryption, Hashing And Other stuff

EnroCrypt This is a Python Module For Encryption, Hashing And Other Basic Stuff You Need, With Secure Encryption And Strong Salted Hashing You Can Do

5 Sep 15, 2022
Tool for working with Y-chromosome data from YFull and FTDNA

ycomp ycomp is a tool for working with Y-chromosome data from YFull and FTDNA. Run ycomp -h for information on how to use the program. Installation Th

Alexander Regueiro 2 Jun 18, 2022
Real-Time and Accurate Full-Body Multi-Person Pose Estimation&Tracking System

News! Aug 2020: v0.4.0 version of AlphaPose is released! Stronger tracking! Include whole body(face,hand,foot) keypoints! Colab now available. Dec 201

Machine Vision and Intelligence Group @ SJTU 6.7k Dec 28, 2022
Pytorch implementation of our paper under review -- 1xN Pattern for Pruning Convolutional Neural Networks

1xN Pattern for Pruning Convolutional Neural Networks (paper) . This is Pytorch re-implementation of "1xN Pattern for Pruning Convolutional Neural Net

Mingbao Lin (林明宝) 29 Nov 29, 2022
exponential adaptive pooling for PyTorch

AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling Abstract Pooling layers are essential building blocks of Convolutional Ne

Alexandros Stergiou 55 Jan 04, 2023
🚗 INGI Dakar 2K21 - Be the first one on the finish line ! 🚗

🚗 INGI Dakar 2K21 - Be the first one on the finish line ! 🚗 This year's first semester Club Info challenge will put you at the head of a car racing

ClubINFO INGI (UCLouvain) 6 Dec 10, 2021
天勤量化开发包, 期货量化, 实时行情/历史数据/实盘交易

TqSdk 天勤量化交易策略程序开发包 TqSdk 是一个由信易科技发起并贡献主要代码的开源 python 库. 依托快期多年积累成熟的交易及行情服务器体系, TqSdk 支持用户使用极少的代码量构建各种类型的量化交易策略程序, 并提供包含期货、期权、股票的 历史数据-实时数据-开发调试-策略回测-

信易科技 2.8k Dec 30, 2022
Non-Attentive-Tacotron - This is Pytorch Implementation of Google's Non-attentive Tacotron.

Non-attentive Tacotron - PyTorch Implementation This is Pytorch Implementation of Google's Non-attentive Tacotron, text-to-speech system. There is som

Jounghee Kim 46 Dec 19, 2022
🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

English | 简体中文 | 繁體中文 State-of-the-art Natural Language Processing for Jax, PyTorch and TensorFlow 🤗 Transformers provides thousands of pretrained mo

Hugging Face 77.2k Jan 02, 2023
PyTorch implementation of Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose

Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose Release Notes The official PyTorch implementation of Neural View S

Angtian Wang 20 Oct 09, 2022
Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

KML: A Machine Learning Framework for Operating Systems & Storage Systems Storage systems and their OS components are designed to accommodate a wide v

File systems and Storage Lab (FSL) 186 Nov 24, 2022
Ensembling Off-the-shelf Models for GAN Training

Data-Efficient GANs with DiffAugment project | paper | datasets | video | slides Generated using only 100 images of Obama, grumpy cats, pandas, the Br

MIT HAN Lab 1.2k Dec 26, 2022
This is a collection of our NAS and Vision Transformer work.

AutoML - Neural Architecture Search This is a collection of our AutoML-NAS work iRPE (NEW): Rethinking and Improving Relative Position Encoding for Vi

Microsoft 832 Jan 08, 2023