Next Best View for Reconstruction

1. Setup Environment

1.1 Install Main Project

mkdir nbv_rec
cd nbv_rec
git clone https://git.hofee.top/hofee/nbv_reconstruction.git

1.2 Install PytorchBoot

the environment is based on PytorchBoot, clone and install it from PytorchBoot

git clone https://git.hofee.top/hofee/PyTorchBoot.git
cd PyTorchBoot
pip install .
cd ..

1.3 Install Blender (Optional)

If you want to render your own dataset as described in section 2. Render Datasets, you'll need to install Blender version 4.0 from Blender Release. Here is an example of installing Blender on Ubuntu:

wget https://download.blender.org/release/Blender4.0/blender-4.0.2-linux-x64.tar.xz
tar -xvf blender-4.0.2-linux-x64.tar.xz

If blender is not in your PATH, you can add it by:

export PATH=$PATH:/path/to/blender/blender-4.0.2-linux-x64

To run the blender script, you need to install the pyyaml and scipy package into your blender python environment. Run the following command to print the python path of your blender:

./blender -b --python-expr "import sys; print(sys.executable)"

Then copy the python path /path/to/blender_python shown in the output and run the following command to install the packages:

/path/to/blender_python -m pip install pyyaml scipy

1.4 Install Blender Render Script (Optional)

Clone the script from nbv_rec_blender_render and rename it to blender:

git clone https://git.hofee.top/hofee/nbv_rec_blender_render.git
mv nbv_rec_blender_render blender

1.5 Check Dependencies

Switch to the project root directory and run pytorch-boot scan or ptb scan to check if all dependencies are installed:

cd nbv_reconstruction
pytorch-boot scan
# or
ptb scan

If you see project structure information in the output, it means all dependencies are correctly installed. Otherwise, you may need to run pip install xxx to install the missing packages.

2. Render Datasets (Optional)

2.1 Download Object Mesh Models

Download the mesh models divided into three parts from:

• object_meshes_part1.zip
• object_meshes_part2.zip
• object_meshes_part3.zip

or download the whole dataset from object_meshes.zip

Download the table model from table.obj

2.2 Set Render Configurations

Open file configs/local/view_generate_config.yaml and modify the parameters to fit your needs. You are required to at least set the following parameters in runner-generate:

• object_dir: the directory of the downloaded object mesh models
• output_dir: the directory to save the rendered dataset
• table_model_path: the path of the downloaded table model

2.3 Render Dataset

There are two ways to render the dataset:

2.3.1 Render with Visual Monitoring

If you want to visually monitor the rendering progress and machine resource usage:

1. In the terminal, run:

   ptb ui
   

2. Open your browser and visit http://localhost:5000
3. Navigate to Project Dashboard - Project Structure - Applications - generate_view
4. Click the Run button to execute the rendering script

2.3.2 Render in Terminal

If you don't need visual monitoring and prefer to run the rendering process directly in the terminal, simply run:

ptb run generate_view

This command will start the rendering process without launching the UI.

3. Preprocess

⚠️ The preprocessing code is currently not managed by PytorchBoot. To run the preprocessing:

1. Open the ./preprocess/preprocessor.py file.

2. Locate the if __name__ == "__main__": block at the bottom of the file.

3. Specify the dataset folder by setting root = "path/to/your/dataset".

4. Run the preprocessing script directly:

   python ./preprocess/preprocessor.py
   

This will preprocess the data in the specified dataset folder.

4. Generate Strategy Label

4.1 Set Configuration

Open the file configs/local/strategy_generate_config.yaml and modify the parameters to fit your needs. You are required to at least set the following parameter:

• datasets.OmniObject3d.root_dir: the directory of your dataset

4.2 Generate Strategy Label

There are two ways to generate the strategy label:

4.2.1 Generate with Visual Monitoring

If you want to visually monitor the generation progress and machine resource usage:

1. In the terminal, run:

   ptb ui
   

2. Open your browser and visit http://localhost:5000
3. Navigate to Project Dashboard - Project Structure - Applications - generate_strategy
4. Click the Run button to execute the generation script

4.2.2 Generate in Terminal

If you don't need visual monitoring and prefer to run the generation process directly in the terminal, simply run:

ptb run generate_strategy

This command will start the strategy label generation process without launching the UI.

5. Train

5.1 Set Configuration

Open the file configs/local/train_config.yaml and modify the parameters to fit your needs. You are required to at least set the following parameters in the experiment section:

experiment:
  name: your_experiment_name
  root_dir: path/to/your/experiment_dir
  use_checkpoint: False # if True, the checkpoint will be loaded
  epoch: 600 # specific epoch to load, -1 stands for last epoch
  max_epochs: 5000 # maximum epochs to train
  save_checkpoint_interval: 1 # save checkpoint interval
  test_first: True # if True, test process will be performed before training at each epoch

Adjust these parameters according to your training requirements.

5.2 Start Training

There are two ways to start the training process:

5.2.1 Train with Visual Monitoring

If you want to visually monitor the training progress and machine resource usage:

1. In the terminal, run:

   ptb ui
   

2. Open your browser and visit http://localhost:5000
3. Navigate to Project Dashboard - Project Structure - Applications - train
4. Click the Run button to start the training process

5.2.2 Train in Terminal

If you don't need visual monitoring and prefer to run the training process directly in the terminal, simply run:

ptb run train

This command will start the training process without launching the UI.

6. Evaluation

...