2021 Real Robot Challenge Phase2 attemp

Overview

Real_Robot_Challenge_Phase2_AE_attemp

We(team name:thriftysnipe) are the first place winner of Phase1 in 2021 Real Robot Challenge.
Please see this page for more details: https://real-robot-challenge.com/leaderboard
To see more details about out Phase1 works: https://github.com/wq13552463699/Real_Robot_challenge
We were granted the access to Phase 2.

I am sorry, the project is too complex with too much large files, It is too hard to upload them all on Github. I just attached a part of the core code here for you to take a quick lreview. If you think my attempts is approriate, you can go to this Google Drive to download the full project file(all codes, results, trained models, environmental files,.etc):
https://drive.google.com/file/d/14vjCrWU6vzMdXxVSR2FeskMvuQpgqWqM/view?usp=sharing

RRC phase2 task description:

Randomly place 25 dices with the size of 0.01x0.01x0.01m in the environment. Use own controller to drive the three-finger robot to rearrange the dice to a specific pattern. Unfortunately, due to the set task is too difficult, no team could complete the task on the actual robot, so all teams with record are awarded third place in this phase. But I think our attempt has a reference value, if later scholars conduct related research, our method may be useful.

Our considerations:

We consider using a reinforcement learning algorithm as the controller in this phase. However, in this phase, information that can play as observations, such as coordinates and orientation of the dices, cannot be obtained from the environment directly but they are crucial for RL to run.
The alternative observations we can use are the images of the three cameras set in 3 different angles in the environment and their segmentation masks. We picked segmentation masks rather than the raw images since the attendance of noise and redundancy in the raw images were too much. Please see the following segmentation mask example(RGB's 3 channels represent segmentation masks from 3 different angles).

The segmentation masks have the dimension of 270x270x3, if directly passing it to the RL agent, which would lead to computational explosion and hard to converge. Hence, we planned to use some means to extract the principal components that can play as observations from it. In addition, the observation value also includes readable read-robot data(joint angle of the robot arm, end effector position, end effector speed, etc.).

Segmentation mask dimensionality reduction

This is the most important part of this task. We tried different methods, such as GAN, VAE, AE, to extract the principal conponents from the images. The quality of data dimensionality reduction can be easily seem from the discripency of reconstructed and oringinal images or the loss curves. After many trials(adjusting hyperparameters, network structure, depth, etc.), we got different trained VAE, GAN and AE models. We conducted offline tests on the obtained model and compared the results, we were surprised to find that the AE performed the best. When the latent of AE is 384, the quality of the reconstructed image is the best. The result is shown in the figure below.

The loss function also converges to an acceptable range:

Build up observation and trian RL agent.

We use the best AE encoder to deal with the segmentation masks to generate the observation and stitch with the readable data. The structure of the overall obervation is shown as follow:
We fed the above observations to several current cutting-edge model based and model free reinforcement learning algorithms, including DDPG+HER, PPO, SLAC, PlaNet and Dreamer. We thought it would work and enable the agent to learn for somewhat anyway. But it is a pity that after many attempts, the model still didn't have any trend to converge. Due to time limited, our attempts were over here.

Some reasons might lead to fail

  1. We used AE as the observation model. Although the AE's dimensionality reduction capability were the best, the latent space of AE were disordered and didn't make sense to RL agent. The observations passed to the RL must be fixed and orderly. Continuous delivery of unfixed data caused a dimensional disaster. For example, the third number in the observation vector passed at t1 represents 'infos of the 1st dice', and the number on the same position at t2 represents the 'infos of the 3rd dice'. This disorderly change with time makes RL very confused.
  2. The extracted latent space from segmentation mask dominates the observations, making RL ignore the existence of robots. The latent space size is 384, but which for the robot data is 27. The two are far apart, and there is a big data bias.
  3. Robot arm blocked the dices, segmentation masks can only represent a part of the dice. This problem cannot be avoided and can only be solved by more powerful image processing technology. This is also a major challenge in the current Image-based RL industry

Contribution

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. Please make sure to update tests as appropriate.

Owner
Qiang Wang
PhD at UCD. Research interest: Reinforcement Learning; Computer vision&Touch; Representation learning
Qiang Wang
GUI wrapper designed for convenient service work with TI CC1352/CC2538/CC2652 based Zigbee sticks or gateways. Packed into single executable file

ZigStar GW Multi tool is GUI wrapper firtsly designed for convenient service work with Zig Star LAN GW, but now supports any TI CC1352/CC2538/CC2652 b

133 Jan 01, 2023
Small Robot, with LIDAR and DepthCamera. Using ROS for Maping and Navigation

🤖 RoboCop 🤖 Small Robot, with LIDAR and DepthCamera. Using ROS for Maping and Navigation Made by Clemente Donoso, 📍 Chile 🇨🇱 RoboCop Lateral Fron

Clemente Donoso Krauss 2 Jan 04, 2022
DNP3 Stalker is a project to analyze and interact with DNP3 devices

DNP3 Stalker Purpose DNP3 Stalker is a project to analyze and interact with DNP3

Cutaway Security, LLC. 2 Feb 10, 2022
Toy robot that traverses on a finite surface

Toy Robot Challenge - Release Notes November 12, 2021 New features Initialisation - Users can set the home position and heading of the robot. Position

Ze Fei Teo 0 Feb 03, 2022
a weather application for the raspberry pi and the Pimorioni Inky pHAT.

raspi-weather a weather application for the raspberry pi and the Inky pHAT

Derek Caelin 59 Oct 24, 2022
ArucoFollow - A script for Robot Operating System and it is a part of a project Robot

ArucoFollow ArucoFollow is a script for Robot Operating System and it is a part

5 Jan 25, 2022
Playing diabolo with two robot arms in ROS + Gazebo

Playing diabolo with robots This repository holds the ROS packages for playing diabolo with two UR5e robot arms on ROS Melodic (Ubuntu 18.04). Read ou

23 Dec 18, 2022
A blender 2.9x addon for managing camera settings

TMG-Camera-Tools A blender 2.9x addon for managing camera settings Tutorial showcasing current features

Mainman002 12 Apr 16, 2022
Create (templateable) cameras that display qr codes in homeassistant

QRCam This custom component creates cameras displaying qrcodes. The QRCodes can be static or generated from templates. If you use a template as conten

Jannes Müller 5 Oct 06, 2022
ENC28J60 Ethernet chip driver for MicroPython (RP2)

micropy-ENC28J60 ENC28J60 Ethernet chip driver for MicroPython v1.17 (RP2) Rationale ENC28J60 is a popular and cheap module for DIY projects. At the m

11 Nov 16, 2022
Trajectory optimization package for Mini-Pupper robot

Trajectory optimization package for Mini-Pupper robot Purpose of this repository is to provide low-torque and low-impact trajectory for Mini-Pupper qu

Sotaro Katayama 38 Aug 17, 2022
CircuitPython library for the CH559 USB to Serial chip

CH559 (USB to Serial) CircuitPython Library Why? Because you might want to get keyboard/mouse/gamepad/HID input into your CircuitPython projects witho

Guy Dupont 3 Nov 19, 2022
Raspberry Pi Pico as a Rubber Ducky

Raspberry-Pi-Pico-as-a-Rubber-Ducky Kurulum Raspberry Pi Pico cihazınız için CircuitPython'u indirin. Boot düğmesine basılı tutarken cihazı bir USB ba

Furkan Enes POLATOĞLU 6 Dec 13, 2022
hardware design of the 250mm drone

hardware design of the 250mm drone

ZJU FAST Lab 645 Dec 25, 2022
Lenovo Legion 5 Pro 2021 Linux RGB Keyboard Light Controller

Lenovo Legion 5 Pro 2021 Linux RGB Keyboard Light Controller This util allows to drive RGB keyboard light on Lenovo Legion 5 Pro 2021 Laptop Requireme

36 Dec 16, 2022
Fener ROS2 package version 2

Fener's ROS2 codes that runs on the vehicle. This node contains basic sensing and actuation nodes for vehicle control. Also example applications will be added.

Muhammed Sezer 1 Jan 18, 2022
Connect a TeslaMate instance to Home Assistant, using MQTT

TeslaBuddy Connect a TeslaMate instance to Home Assistant, using MQTT. It allows basic control of your Tesla vehicle via Home Assistant (currently, ju

4 May 23, 2022
Play music on Raspberry Pi Pico Without CPU involvement

MicroPython_PIO_Music_DMA Play music on Raspberry Pi Pico Without CPU involvement This is based on PIOBeep (https://github.com/benevpi/pico_pio_buzz)

3 Nov 27, 2022
E-Ink Magic Calendar that automatically syncs to Google Calendar and runs off a battery powered Raspberry Pi Zero

E-Ink Magic Calendar that automatically syncs to Google Calendar and runs off a battery powered Raspberry Pi Zero

2.8k Dec 30, 2022
A Home Assistant sensor that tells you what holiday is next

Next Holiday Sensor This sensor tells you what holiday is coming up next. You can use it to set holiday light colors or other scenes. The state of the

Nick Touran 20 Dec 04, 2022