Script for resizing MTD partitions on a QNAP device in order to be available to upgrade from buster to bullseye

Overview

QNAP partitions resize for kirkwood devices.

As explained by Marin Michlmayr, Debian bullseye support on kirkwood QNAP devices was dropped due to [mainly] the limited size of the Kernel partition (2MB).

Indeed, Bullseye current kernel image (vmlinuz-5.10.0-8-marvell) is 2445216 bytes long (2.3MB)

In addition, partition for initrd is also limited (9MB) which may lead to space issues.

Hopefully, some space is still unused for Debian in QNAP 16MB NOR flash.

  • An additional 3MB Rootfs2 partition is used by original QNAP firmware for its own purpose (install on empty HDD ?)
  • A "NAS config" partition is 1.2MB large despite containing few configuration files (<128KB). This partition can be resized to 256KB (Flash block size) without losing the information.

New Layout

With this script, we propose to use a new flash layout

Layout

  • We keep /dev/mtdX numbers mapping, in case some other users are using a fix numbering.
  • we keep a window on legacy kernel mapping to help if we want to restore the original QNAP firmware or to install the Buster installer
  • Rootfs1 is larger but use the same start offset (simplify the transition since no write in flash is required)
  • Kernel is larger. We must be careful during the transition since offsets are different.

With this new layout, we can transparently upgrade to Bullseye

  • More room for kernel and initrd
  • Future kernel updates performed during apt upgradewill use the new layout without any further change or manual operations.

Kernel and MTD partitions

Linux has 2 methods for configuring the partitions.

  1. Device Tree

    This is the standard way to describe the device.

    see: https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/kirkwood-ts219.dtsi

    ; label = "Kernel"; }; [email protected] { reg = <0x00400000 0x00900000>; label = "RootFS1"; }; [email protected] { reg = <0x00d00000 0x00300000>; label = "RootFS2"; }; [email protected] { reg = <0x00080000 0x00040000>; label = "U-Boot Config"; }; [email protected] { reg = <0x000c0000 0x00140000>; label = "NAS Config"; }; }; ">
    			[email protected] {
    				[...]
    				[email protected] {
    					reg = <0x00000000 0x00080000>;
    					label = "U-Boot";
    				};
    				[email protected] {
    					reg = <0x00200000 0x00200000>;
    					label = "Kernel";
    				};
    				[email protected] {
    					reg = <0x00400000 0x00900000>;
    					label = "RootFS1";
    				};
    				[email protected] {
    					reg = <0x00d00000 0x00300000>;
    					label = "RootFS2";
    				};
    				[email protected] {
    					reg = <0x00080000 0x00040000>;
    					label = "U-Boot Config";
    				};
    				[email protected] {
    					reg = <0x000c0000 0x00140000>;
    					label = "NAS Config";
    				};
    			};
    

    On Debian, binary versions (dtb) are provided by the linux-image-4.19.0-16-marvell package.

    Every time a new kernel version is flashed by flash-kernel, the proper dtb blob is concatenated to the kernel image so the kernel knows the details of the machine and is able to configure all the drivers properly (including the MTD partitions).

    It is not difficult to build a DTB and configure Debian to use this alternate DTB file. It simply need to be present in /etc/flash-kernel/dtbs/directory and flash-kernel will correctly switch to our own specific file.

    Example:

    cd [clone of linux sources]/linux
    # modify arch/arm/boot/dts/kirkwood-ts219.dtsi
    
    for D in kirkwood-ts219-6281 kirkwood-ts219-6282; do
        cpp -nostdinc -I include -I arch  -undef -x assembler-with-cpp \
           ./arch/arm/boot/dts/$D.dts \
           /tmp/preprocess.dts
        dtc -O dtb -o /tmp/$D.dtb /tmp/preprocess.dts
        cp /tmp/$D.dtb /etc/flash-kernel/dtbs/
        done
    
    
  2. kernel boot cmdline

    U-boot can also override the DTB information by using mtdparts=....options as parsed by https://github.com/torvalds/linux/blob/master/drivers/mtd/parsers/cmdlinepart.c

    The kernel try use use cmdline parameters before DTB information.

    Original QNAP U-boot configuration doesn't use this method.

I select the "kernel boot cmdline" solution to configure the new layout:

  • if /etc/flash-kernel/dtbs/content is modified or erased for some reasons (new install ?) the u-boot setup and the kernel MTD usage will not be synchronized and the device will fail to boot.
  • cmdline only "patch/override" the MTD partitions information. If the original DTB file provided by Debian is updated for some reasons (driver fixes), the kernel will still continue to use those fixes.

The qnap_mtd_resize.py script will:

  • Resize the "NAS config" filesystem
  • Prepare the content of current "NAS config" partition (offset 0xc0000 to 0x200000) with the shrink FS + first 1MB of the current kernel
  • prepare the image of current "Kernel" partition (starting at 0x200000) with tail of the current kernel
  • Patch U-boot env/config for the new bootargsand bootcmdvariables.
  • Write flash from 0xc0000 to 0x200000 ("NAS config" + 1MB head of kernel)
  • write flash from 0x200000 with 1MB tail of kernel.

Resize process

First, Do a backup of your MTD

cat /dev/mtd0 > /tmp/mtd0.uboot.backup
cat /dev/mtd1 > /tmp/mtd1.kernel.backup
cat /dev/mtd2 > /tmp/mtd2.rootfs1.backup
cat /dev/mtd3 > /tmp/mtd3.rootfs2.backup
cat /dev/mtd4 > /tmp/mtd4.uboot-config.backup
cat /dev/mtd5 > /tmp/mtd5.nas-config.backup
cd /tmp
tar cvzf mtd_backup.tgz mtd?.*.backup

And save this mtd_backup.tgzon your PC, transfering the file with scp / sftp or a USB drive....

Then, run qnap_mtd_resize.py

A first run with --dry-runoption to check that everything will be fine (except flashing)

sudo ./qnap_mtd_resize.py --dry-run

Example of dry-run log here.

If everything is fine run again without --dry-run

sudo ./qnap_mtd_resize.py

And reboot...

You are now running the same system, but with more room:

$ cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00080000 00040000 "uboot"
mtd1: 00300000 00040000 "Kernel"
mtd2: 00c00000 00040000 "RootFS1"
mtd3: 00200000 00040000 "Kernel_legacy"
mtd4: 00040000 00040000 "U-Boot Config"
mtd5: 00040000 00040000 "NAS Config"

Which makes possible to install Bullseye's kernel:

$ flash-kernel 
kirkwood-qnap: machine: QNAP TS219 family
Using DTB: kirkwood-ts219-6281.dtb
Installing /usr/lib/linux-image-5.10.0-8-marvell/kirkwood-ts219-6281.dtb into /boot/dtbs/5.10.0-8-marvell/./kirkwood-ts219-6281.dtb
Taking backup of kirkwood-ts219-6281.dtb.
Installing new kirkwood-ts219-6281.dtb.
flash-kernel: installing version 5.10.0-8-marvell
flash-kernel: appending /usr/lib/linux-image-5.10.0-8-marvell/kirkwood-ts219-6281.dtb to kernel
Generating kernel u-boot image... done.
Flashing kernel (using 2455558/3145728 bytes)... done.
Flashing initramfs (using 3992060/12582912 bytes)... done.

Additional configuration to improveinitrd size

Even if we increase Rootfs1 from 9 to 12 MB, you can still decrease the initrd size by compressing with xz

/etc/initramfs-tools/conf.d/compress ">
echo "COMPRESS=xz" > /etc/initramfs-tools/conf.d/compress

List of tested devices:

Model cat /sys/firmware/devicetree/base/model DTB file uboot env
(legacy)
uboot_env
(new)
Resize log
TS-219P QNAP TS219 family kirkwood-ts219-6281.dtb
TS-419PII QNAP TS419 family kirkwood-ts419-6282.dtb QNAP_TS419_family,uboot-env.legacy QNAP_TS419_family,uboot-env.new log
Owner
Arnaud Mouiche
Arnaud Mouiche
Demo Python project using Conda and Poetry

Conda Poetry This is a demonstration of how Conda and Poetry can be used in a Python project for dev dependency management and production deployment.

Ryan Allen 2 Apr 26, 2022
Find the remote website version based on a git repository

versionshaker Versionshaker is a tool to find a remote website version based on a git repository This tool will help you to find the website version o

Orange Cyberdefense 110 Oct 23, 2022
A replacement of qsreplace, accepts URLs as standard input, replaces all query string values with user-supplied values and stdout.

Bhedak A replacement of qsreplace, accepts URLs as standard input, replaces all query string values with user-supplied values and stdout. Works on eve

Eshan Singh 84 Dec 31, 2022
Verification of Monty Hall problem by experimental simulation.

Verification of Monty Hall problem by experimental simulation. |中文|English| In the process of learning causal inference, I learned about the Monty Hal

云端听茗 1 Nov 22, 2022
VCM EE1.2 P-layer feature map anchor generation 137th MPEG-VCM

VCM EE1.2 P-layer feature map anchor generation 137th MPEG-VCM

IPSL 6 Oct 18, 2022
Imports an object based on a string import_string('package.module:function_name')() - Based on werkzeug.utils

DEPRECATED don't use it. Please do: import importlib foopath = 'src.apis.foo.Foo' module_name = '.'.join(foopath.split('.')[:-1]) # to get src.apis.f

Bruno Rocha Archived Projects 11 Nov 12, 2022
ThnoolBox - A thneed is a multi-use versatile object

ThnoolBox Have you ever wanted a collection of bodged desktop apps that are Lorax themed ? No ? Sucks to suck I guess Apps & their downsides CalculaTh

pocoyo 1 Jan 21, 2022
A collection of python exercises to help your learning path!

How to use Step 1: run this command git clone https://github.com/TechPenguineer/Python-Exercises.git Step 2: Run this command cd Python-Exercises You

Tech Penguin 5 Aug 05, 2021
uMap lets you create maps with OpenStreetMap layers in a minute and embed them in your site.

uMap project About uMap lets you create maps with OpenStreetMap layers in a minute and embed them in your site. Because we think that the more OSM wil

771 Dec 29, 2022
Submission from Team OMR for the TRI-NIT Hackathon

Submission from Team OMR for the TRI-NIT Hackathon

0 Feb 01, 2022
GitHub Actions Version Updater Updates All GitHub Action Versions in a Repository and Creates a Pull Request with the Changes.

GitHub Actions Version Updater GitHub Actions Version Updater is GitHub Action that is used to update other GitHub Actions in a Repository and create

Maksudul Haque 42 Dec 22, 2022
IOP Support for Python (Experimental)

TAGS Experimental IOP Framework for Python WARNING: Currently, this project has NO EXCEPTION HANDLING. USE AT YOUR OWN RISK! I. Introduction to Interf

1 Oct 22, 2021
Automatically give thanks to Pypi packages you use in your project!

Automatically give thanks to Pypi packages you use in your project!

Ward 25 Dec 20, 2021
PyScaffold is a project generator for bootstrapping high quality Python packages

PyScaffold is a project generator for bootstrapping high quality Python packages, ready to be shared on PyPI and installable via pip. It is easy to use and encourages the adoption of the best tools a

PyScaffold 1.7k Jan 03, 2023
A faster copy of nell's comet nuker

Astro a faster copy of nell's comet nuker also nell uses external libraries like it's cocaine man never learned to use ansi color codes (ily nell) (On

horrid 8 Aug 15, 2022
Demo code for "Logs in distributed systems" webinar

Hexlet Logs Demo Пререквизиты docker-compose python3 Учетка в DataDog Базовое понимание, что такое логи (можно почитать гайд

Anton Markelov 1 Dec 01, 2021
Data Poisoning based on Adversarial Attacks using Non-Robust Features

Data Poisoning based on Adversarial Attacks using Non-Robust Features Usage python main.py [-h] [--gpu | -g GPU] [--eps |-e EPSILON] [--pert | -p PER

Jonathan E. 1 Nov 02, 2021
Morth - Stack Based Programming Language

Morth WARNING! THIS LANGUAGE IS A WORKING PROGRESS. THIS IS JUST A HOBBY PROJECT

Dominik Danner 2 Mar 05, 2022
Pytorch implementation of "Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates"

Peer Loss functions This repository is the (Multi-Class & Deep Learning) Pytorch implementation of "Peer Loss Functions: Learning from Noisy Labels wi

Kushal Shingote 1 Feb 08, 2022
Discord's own Dumbass made for shits n' Gigs!

FWB3 Discord's own Dumbass made for shits n' Gigs! Please note: This bot is made to be stupid and funny, If you want to get into bot development you'r

1 Dec 06, 2021