146 lines
4.1 KiB
Markdown
Executable File
146 lines
4.1 KiB
Markdown
Executable File
# ActivePerception
|
|
## 1 Installation
|
|
### 1.1 Requirements
|
|
- **Operating System**: Linux Only. We test this project on Ubuntu 22.04.
|
|
- **Cuda Toolkit Version**: cuda 11.8 or higher.
|
|
- **Python Version**: python3.9 or higher
|
|
- **Pytorch Version**: Pytorch 2.0.0+cu118
|
|
|
|
### 1.2 Install ActivePerception Environment
|
|
#### 1.2.1 Install Python and `requirements.txt`
|
|
Clone this repo with
|
|
```
|
|
git clone https://github.com/Jiyao06/ActivePerception.git
|
|
```
|
|
Go to repo root directory
|
|
```
|
|
cd ActivePerception
|
|
```
|
|
Create python Environment
|
|
```
|
|
conda create --name nbv python==3.9
|
|
```
|
|
Install the basic environment with `requirements.txt`
|
|
```
|
|
pip install -r requirements.txt
|
|
```
|
|
> if you encounter this issue while running the project: `ImportError: cannot import name 'NDArray' from 'numpy.typing'`, please upgrade numpy to version 1.23.5 with `pip install numpy==1.23.5`
|
|
#### 1.2.2 Install pointnet++ for ActivePerception
|
|
Go to pointnet2 repo root directory
|
|
```
|
|
cd modules/module_lib/pointnet2_utils/pointnet2
|
|
```
|
|
Install pointnet2
|
|
```
|
|
pip install -e .
|
|
```
|
|
|
|
|
|
### 1.3 Install GSNet Environment for Grasping Evaluation **(Optional)**
|
|
#### 1.3.1 Install MinkowskiEngine (Linux Only)
|
|
Install dependencies of MinkowskiEngine
|
|
```
|
|
sudo apt-get update
|
|
sudo apt-get install -y git ninja-build cmake build-essential libopenblas-dev xterm xauth openssh-server tmux wget mate-desktop-environment-core
|
|
```
|
|
Clone MinkowskiEngine git repo
|
|
```
|
|
cd <any_path>
|
|
git clone --recursive "https://github.com/NVIDIA/MinkowskiEngine"
|
|
```
|
|
Install MinkowskiEngine
|
|
```
|
|
cd MinkowskiEngine
|
|
python setup.py install --force_cuda --blas=openblas
|
|
```
|
|
|
|
### 1.3.2 Install pointnet++ for GSNet
|
|
Go to GSNet repo root
|
|
```
|
|
cd <your_ActivePerception_repo_root>/baselines/grasping/GSNet
|
|
```
|
|
Go to pointnet++ repo root and install it
|
|
```
|
|
cd pointnet2
|
|
python setup.py install
|
|
```
|
|
|
|
### 1.3.3 Install KNN operator
|
|
```
|
|
cd ../knn
|
|
python setup.py install
|
|
```
|
|
|
|
### 1.3.4 Install GraspnetAPI for Evaluation
|
|
```
|
|
cd ../graspnetAPI
|
|
pip install .
|
|
```
|
|
|
|
### 1.4 Install FoundationPose Environment for Object Pose Evaluation **(Optional)**
|
|
...
|
|
|
|
## 2 Training
|
|
### 2.1 Prapare Datasets
|
|
Please download the dataset from the links shown as below:
|
|
- [NBV Simulation (Train Dataset)](https://link-url-here.org)
|
|
- [NBV Simulation (Test Dataset)](https://link-url-here.org)
|
|
- [NBV Simulation (Object Models)](https://link-url-here.org)
|
|
|
|
or directly download a pre-organized dataset structure to skip "2.2 Organize Dataset Structure"
|
|
- [NBV Simulation (Pre-organized Dataset)](https://link-url-here.org)
|
|
|
|
### 2.2 Organize Dataset Structure
|
|
Please organize the dataset into the following structure:
|
|
```
|
|
$ROOT_OF_DATA
|
|
- objects
|
|
- obj_name_0
|
|
- obj_name_1
|
|
- ...
|
|
- train
|
|
- scene_name_0
|
|
- scene_name_1
|
|
- ...
|
|
- test
|
|
- scene_name_0
|
|
- scene_name_1
|
|
- ...
|
|
```
|
|
### 2.3 Training for Grasping
|
|
#### 2.3.1 Prepare Pretrained GSNet weights
|
|
Please download the pretrained GSNet weights from: [pretrained_gsnet_weights](https://link-url-here.org)
|
|
#### 2.3.2 Preprocess View Score
|
|
...
|
|
|
|
#### 2.3.3 Training Configuration
|
|
Open training config file in `ActivePerception/configs/server_train_config.yaml`.
|
|
|
|
To run the training task, you need to customize at least the following experiment configuration:
|
|
- **experiment name** at `experiment -> name`
|
|
- **grasp pretrained model path** at `experiment -> grasp_model_path`
|
|
- **dataset root path** at `datasets -> general -> data_dir`
|
|
|
|
#### 2.3.4 Run View Generator Web Server
|
|
In order to test the model's predictions under the new view after each epoch's training, a view generator is needed. Therefore, you need to run the following command to start it.
|
|
|
|
Otherwise, please comment out all instances of the `grasp_improvement` method for the test set in `server_train_config.yaml` under `settings -> test -> dataset_list -> eval_list`. And then you won't get the result of grasp score's improvement during training.
|
|
|
|
```
|
|
python runners/view_generator.py
|
|
```
|
|
|
|
#### 2.3.5 Start Training
|
|
Run following command to start the training task.
|
|
```
|
|
python runners/trainer.py
|
|
```
|
|
|
|
## 3 Evaluation
|
|
...
|
|
|
|
## 4 Data and Results Visualization
|
|
Vue.js framework is required to ...
|
|
|
|
## 5 Custom Sim Data Generation
|
|
... |