Script for resizing MTD partitions on a QNAP device in order to be available to upgrade from buster to bullseye

Overview

QNAP partitions resize for kirkwood devices.

As explained by Marin Michlmayr, Debian bullseye support on kirkwood QNAP devices was dropped due to [mainly] the limited size of the Kernel partition (2MB).

Indeed, Bullseye current kernel image (vmlinuz-5.10.0-8-marvell) is 2445216 bytes long (2.3MB)

In addition, partition for initrd is also limited (9MB) which may lead to space issues.

Hopefully, some space is still unused for Debian in QNAP 16MB NOR flash.

  • An additional 3MB Rootfs2 partition is used by original QNAP firmware for its own purpose (install on empty HDD ?)
  • A "NAS config" partition is 1.2MB large despite containing few configuration files (<128KB). This partition can be resized to 256KB (Flash block size) without losing the information.

New Layout

With this script, we propose to use a new flash layout

Layout

  • We keep /dev/mtdX numbers mapping, in case some other users are using a fix numbering.
  • we keep a window on legacy kernel mapping to help if we want to restore the original QNAP firmware or to install the Buster installer
  • Rootfs1 is larger but use the same start offset (simplify the transition since no write in flash is required)
  • Kernel is larger. We must be careful during the transition since offsets are different.

With this new layout, we can transparently upgrade to Bullseye

  • More room for kernel and initrd
  • Future kernel updates performed during apt upgradewill use the new layout without any further change or manual operations.

Kernel and MTD partitions

Linux has 2 methods for configuring the partitions.

  1. Device Tree

    This is the standard way to describe the device.

    see: https://github.com/torvalds/linux/blob/master/arch/arm/boot/dts/kirkwood-ts219.dtsi

    ; label = "Kernel"; }; [email protected] { reg = <0x00400000 0x00900000>; label = "RootFS1"; }; [email protected] { reg = <0x00d00000 0x00300000>; label = "RootFS2"; }; [email protected] { reg = <0x00080000 0x00040000>; label = "U-Boot Config"; }; [email protected] { reg = <0x000c0000 0x00140000>; label = "NAS Config"; }; }; ">
    			[email protected] {
    				[...]
    				[email protected] {
    					reg = <0x00000000 0x00080000>;
    					label = "U-Boot";
    				};
    				[email protected] {
    					reg = <0x00200000 0x00200000>;
    					label = "Kernel";
    				};
    				[email protected] {
    					reg = <0x00400000 0x00900000>;
    					label = "RootFS1";
    				};
    				[email protected] {
    					reg = <0x00d00000 0x00300000>;
    					label = "RootFS2";
    				};
    				[email protected] {
    					reg = <0x00080000 0x00040000>;
    					label = "U-Boot Config";
    				};
    				[email protected] {
    					reg = <0x000c0000 0x00140000>;
    					label = "NAS Config";
    				};
    			};
    

    On Debian, binary versions (dtb) are provided by the linux-image-4.19.0-16-marvell package.

    Every time a new kernel version is flashed by flash-kernel, the proper dtb blob is concatenated to the kernel image so the kernel knows the details of the machine and is able to configure all the drivers properly (including the MTD partitions).

    It is not difficult to build a DTB and configure Debian to use this alternate DTB file. It simply need to be present in /etc/flash-kernel/dtbs/directory and flash-kernel will correctly switch to our own specific file.

    Example:

    cd [clone of linux sources]/linux
    # modify arch/arm/boot/dts/kirkwood-ts219.dtsi
    
    for D in kirkwood-ts219-6281 kirkwood-ts219-6282; do
        cpp -nostdinc -I include -I arch  -undef -x assembler-with-cpp \
           ./arch/arm/boot/dts/$D.dts \
           /tmp/preprocess.dts
        dtc -O dtb -o /tmp/$D.dtb /tmp/preprocess.dts
        cp /tmp/$D.dtb /etc/flash-kernel/dtbs/
        done
    
    
  2. kernel boot cmdline

    U-boot can also override the DTB information by using mtdparts=....options as parsed by https://github.com/torvalds/linux/blob/master/drivers/mtd/parsers/cmdlinepart.c

    The kernel try use use cmdline parameters before DTB information.

    Original QNAP U-boot configuration doesn't use this method.

I select the "kernel boot cmdline" solution to configure the new layout:

  • if /etc/flash-kernel/dtbs/content is modified or erased for some reasons (new install ?) the u-boot setup and the kernel MTD usage will not be synchronized and the device will fail to boot.
  • cmdline only "patch/override" the MTD partitions information. If the original DTB file provided by Debian is updated for some reasons (driver fixes), the kernel will still continue to use those fixes.

The qnap_mtd_resize.py script will:

  • Resize the "NAS config" filesystem
  • Prepare the content of current "NAS config" partition (offset 0xc0000 to 0x200000) with the shrink FS + first 1MB of the current kernel
  • prepare the image of current "Kernel" partition (starting at 0x200000) with tail of the current kernel
  • Patch U-boot env/config for the new bootargsand bootcmdvariables.
  • Write flash from 0xc0000 to 0x200000 ("NAS config" + 1MB head of kernel)
  • write flash from 0x200000 with 1MB tail of kernel.

Resize process

First, Do a backup of your MTD

cat /dev/mtd0 > /tmp/mtd0.uboot.backup
cat /dev/mtd1 > /tmp/mtd1.kernel.backup
cat /dev/mtd2 > /tmp/mtd2.rootfs1.backup
cat /dev/mtd3 > /tmp/mtd3.rootfs2.backup
cat /dev/mtd4 > /tmp/mtd4.uboot-config.backup
cat /dev/mtd5 > /tmp/mtd5.nas-config.backup
cd /tmp
tar cvzf mtd_backup.tgz mtd?.*.backup

And save this mtd_backup.tgzon your PC, transfering the file with scp / sftp or a USB drive....

Then, run qnap_mtd_resize.py

A first run with --dry-runoption to check that everything will be fine (except flashing)

sudo ./qnap_mtd_resize.py --dry-run

Example of dry-run log here.

If everything is fine run again without --dry-run

sudo ./qnap_mtd_resize.py

And reboot...

You are now running the same system, but with more room:

$ cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00080000 00040000 "uboot"
mtd1: 00300000 00040000 "Kernel"
mtd2: 00c00000 00040000 "RootFS1"
mtd3: 00200000 00040000 "Kernel_legacy"
mtd4: 00040000 00040000 "U-Boot Config"
mtd5: 00040000 00040000 "NAS Config"

Which makes possible to install Bullseye's kernel:

$ flash-kernel 
kirkwood-qnap: machine: QNAP TS219 family
Using DTB: kirkwood-ts219-6281.dtb
Installing /usr/lib/linux-image-5.10.0-8-marvell/kirkwood-ts219-6281.dtb into /boot/dtbs/5.10.0-8-marvell/./kirkwood-ts219-6281.dtb
Taking backup of kirkwood-ts219-6281.dtb.
Installing new kirkwood-ts219-6281.dtb.
flash-kernel: installing version 5.10.0-8-marvell
flash-kernel: appending /usr/lib/linux-image-5.10.0-8-marvell/kirkwood-ts219-6281.dtb to kernel
Generating kernel u-boot image... done.
Flashing kernel (using 2455558/3145728 bytes)... done.
Flashing initramfs (using 3992060/12582912 bytes)... done.

Additional configuration to improveinitrd size

Even if we increase Rootfs1 from 9 to 12 MB, you can still decrease the initrd size by compressing with xz

/etc/initramfs-tools/conf.d/compress ">
echo "COMPRESS=xz" > /etc/initramfs-tools/conf.d/compress

List of tested devices:

Model cat /sys/firmware/devicetree/base/model DTB file uboot env
(legacy)
uboot_env
(new)
Resize log
TS-219P QNAP TS219 family kirkwood-ts219-6281.dtb
TS-419PII QNAP TS419 family kirkwood-ts419-6282.dtb QNAP_TS419_family,uboot-env.legacy QNAP_TS419_family,uboot-env.new log
Owner
Arnaud Mouiche
Arnaud Mouiche
CNKD - Minimalistic Windows ransomware written in Python

CNKD Minimalistic Windows ransomware written in Python (Still a work in progress

Alex 2 May 27, 2022
A python script that will automate the boring task of login to the captive portal again and again

A python script that will automate the boring task of login to the captive portal again and again

Rakib Hasan 2 Feb 09, 2022
Twikoo自定义表情列表 | HexoPlusPlus自定义表情列表(其实基于OwO的项目都可以用的啦)

Twikoo-Magic 更新说明 2021/1/15 基于2021/1/14 Twikoo 更新1.1.0-beta,所有表情都将以缩写形式(如:[ text ]:)输出。1/14之前本仓库有部分表情text缺失及重复, 导致无法正常使用表情 1/14后的所有表情json列表已全部更新

noionion 90 Jan 05, 2023
Arabic to Roman Converter in Python

Arabic-to-Roman-Converter Made together with https://github.com/goltaraya . Arabic to Roman Converter in Python. -Instructions: 1 - Make sure you have

Pedro Lucas Tomazeti Fernandes 6 Oct 28, 2021
Shopify Backend Developer Intern Challenge - Summer 2022

Shopify Backend Developer Intern The task is build an inventory tracking web application for a logistics company. The detailed task details can be fou

Meet Gandhi 11 Oct 08, 2022
Team10 backend - A service which accepts a VRM (Vehicle Registration Mark)

GreenShip - API A service which accepts a VRM (Vehicle Registration Mark) and re

3D Hack 1 Jan 21, 2022
Python 3 script for installing kali tools on your linux machine

Python 3 script for installing kali tools on your linux machine

gh0st 2 Apr 20, 2022
Practice in Oxford_AI&ML class

Practice in Oxford_AI&ML class

St3ve Lee 2 Feb 04, 2022
Sudo type me a payload

payloadSecretary Sudo type me a payload Have you ever found yourself having to perform a test, and a client has provided you with a VM inside a VDI in

7 Jul 21, 2022
LPCV Winner Solution of Spring Team

LPCV Winner Solution of Spring Team

22 Jul 20, 2022
Demo content - Automate your automation!

Automate-AAP2 Demo Content - Automate your automation! A fully automated Ansible Automation Platform. Context Installing and configuring Ansible Autom

0 Oct 27, 2022
Research on how Gboard Stickers work.

Google-Sticker-Mashup-Research Research on how Gboard Stickers work. Contribute Contributing is nice, and you will be listed below for contributing. C

Jeremiah 45 Oct 28, 2022
System Design Assignments as part of Arpit's System Design Masterclass

System Design Assignments The repository contains a set of problem statements around Software Architecture and System Design as conducted by Arpit's S

Relog 1.1k Jan 09, 2023
Simple script with AminoLab to send ghost messages

Simple script with AminoLab to send ghost messages

Moleey 1 Nov 22, 2021
Add any Program in any language you like or add a hello world Program ❣️ if you like give us :star:

Welcome to the Hacktoberfest 2018 Hello-world 📋 This Project aims to help you to get started with using Github. You can find a tutorial here What is

Aniket Sharma 1.5k Nov 16, 2022
Code for the manim-generated scenes used in 3blue1brown videos

This project contains the code used to generate the explanatory math videos found on 3Blue1Brown. This almost entirely consists of scenes generated us

Grant Sanderson 4.1k Jan 02, 2023
MIXLAB_NASA_TICKET mixlab 灵感来源于NASA的火星船票

MIXLAB_NASA_TICKET mixlab 灵感来源于NASA的火星船票,我们想要使用开源的代码来定制化这一设计。 其中photo_to_cartoon 是paddle的开源代码:https://github.com/minivision-ai/photo2cartoon-paddle 也借

tongji_cy 38 Feb 20, 2022
Multiperiod Reports by Month/Quarter/Year in Beancount.

Multiperiod Reports by Month/Quarter/Year in Beancount. Plotting income and expenses over time. Treemap plot of expenses.

Altynbek Isabekov 16 Aug 13, 2022
An ongoing curated list of frameworks, libraries, learning tutorials, software and resources in Python Language.

Python Development Welcome to the world of Python. An ongoing curated list of frameworks, libraries, learning tutorials, software and resources in Pyt

Paul Veillard 2 Dec 24, 2021
Improved version calculator, now using while True and etc

CalcuPython_2.0 Olá! Calculadora versão melhorada, agora usando while True e etc... melhorei o design e os carai tudo (rode no terminal, pra melhor ex

Scott 2 Jan 27, 2022