M3DM的autodl环境构建过程笔记
文章目录
- 在3D-ADS环境https://blog.csdn.net/tfxzgp/article/details/144259472基础上构建M3DM(失败的记录，不用看)
- 更换镜像重来（成功）
- 安装缺少的包
- 修改models.py中的RGB和点云backbone的路径
- 修改main.py路径参数
- 运行
在3D-ADS环境https://blog.csdn.net/tfxzgp/article/details/144259472基础上构建M3DM(失败的记录，不用看)
conda deactivate
conda create -n m3dmfrombase --clone base
conda create --name zgp_ader --clone zgp_3dads
conda activate m3dmfrombase
(base) root@autodl-container-a3574a91f4-10f31e68:~# conda deactivate
root@autodl-container-a3574a91f4-10f31e68:~# conda activate m3dmfrombase
(m3dmfrombase) root@autodl-container-a3574a91f4-10f31e68:~# python /tmp/3D-ADS/main.py
(m3dmfrombase) root@autodl-container-a3574a91f4-10f31e68:~# python -m pip install /tmp/Pointnet2_PyTorch-master/pointnet2_ops_lib
ERROR: Command errored out with exit status 1: /root/miniconda3/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-j0y4rde9/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-j0y4rde9/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-u1hg9uzb/install-record.txt --single-version-externally-managed --compile --install-headers /root/miniconda3/include/python3.8/pointnet2-ops Check the logs for full command output.
也许是自己的资源不够，有卡开机再次尝试
(m3dmfrombase) root@autodl-container-9eae489b28-92e975dd:~# python -m pip install /tmp/Pointnet2_PyTorch-master/pointnet2_ops_lib
仍然
e-packages/torch/include/TH -I/root/miniconda3/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/miniconda3/include/python3.8 -c pointnet2_ops/_ext-src/src/sampling_gpu.cu -o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/sampling_gpu.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -O3 -Xfatbin -compress-all -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_37,code=compute_37 -gencode=arch=compute_37,code=sm_37 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_62,code=sm_62 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -std=c++14nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).g++ -pthread -shared -B /root/miniconda3/compiler_compat -L/root/miniconda3/lib -Wl,-rpath=/root/miniconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/ball_query.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/bindings.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/group_points.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/interpolate.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/sampling.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/ball_query_gpu.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/group_points_gpu.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/interpolate_gpu.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/sampling_gpu.o -L/root/miniconda3/lib/python3.8/site-packages/torch/lib -L/usr/local/cuda/lib64 -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda_cu -ltorch_cuda_cpp -o build/lib.linux-x86_64-3.8/pointnet2_ops/_ext.cpython-38-x86_64-linux-gnu.so/root/miniconda3/compiler_compat/ld: cannot find /lib64/libpthread.so.0/root/miniconda3/compiler_compat/ld: cannot find /usr/lib64/libpthread_nonshared.acollect2: error: ld returned 1 exit statuserror: command 'g++' failed with exit status 1
。。。。。。
ERROR: Command errored out with exit status 1: /root/miniconda3/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1ur90alg/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1ur90alg/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8o0w291z/install-record.txt --single-version-externally-managed --compile --install-headers /root/miniconda3/include/python3.8/pointnet2-ops Check the logs for full command output.发现克隆的环境下没有自己的python。
在原环境下修改
(base) root@autodl-container-9eae489b28-92e975dd:~# python -m pip install /tmp/Pointnet2_PyTorch-master/pointnet2_ops_lib
g++ -pthread -shared -B /root/miniconda3/compiler_compat -L/root/miniconda3/lib -Wl,-rpath=/root/miniconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/ball_query.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/bindings.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/group_points.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/interpolate.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/sampling.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/ball_query_gpu.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/group_points_gpu.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/interpolate_gpu.o build/temp.linux-x86_64-3.8/pointnet2_ops/_ext-src/src/sampling_gpu.o -L/root/miniconda3/lib/python3.8/site-packages/torch/lib -L/usr/local/cuda/lib64 -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda_cu -ltorch_cuda_cpp -o build/lib.linux-x86_64-3.8/pointnet2_ops/_ext.cpython-38-x86_64-linux-gnu.so/root/miniconda3/compiler_compat/ld: cannot find /lib64/libpthread.so.0/root/miniconda3/compiler_compat/ld: cannot find /usr/lib64/libpthread_nonshared.acollect2: error: ld returned 1 exit statuserror: command 'g++' failed with exit status 1一样
vim ~/.zshrc
pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
source ~/.zshrc
失败
更换镜像重来（成功）
source ~/.bashrc
conda env list
# conda environments:
#
base * /root/miniconda3(base)conda list
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 4.5 1_gnu defaults
absl-py 1.1.0 pypi_0 pypi
addict 2.4.0 pypi_0 pypi
anyio 3.6.1 pypi_0 pypi
argon2-cffi 21.3.0 pypi_0 pypi
argon2-cffi-bindings 21.2.0 pypi_0 pypi
asttokens 2.0.5 pypi_0 pypi
attrs 21.4.0 pypi_0 pypi
babel 2.10.3 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
beautifulsoup4 4.11.1 pypi_0 pypi
black 22.6.0 pypi_0 pypi
bleach 5.0.0 pypi_0 pypi
brotlipy 0.7.0 py38h27cfd23_1003 defaults
ca-certificates 2021.7.5 h06a4308_1 defaults
cachetools 5.2.0 pypi_0 pypi
certifi 2021.5.30 py38h06a4308_0 defaults
cffi 1.14.6 py38h400218f_0 defaults
chardet 4.0.0 py38h06a4308_1003 defaults
click 8.1.3 pypi_0 pypi
colorama 0.4.5 pypi_0 pypi
commonmark 0.9.1 pypi_0 pypi
conda 4.10.3 py38h06a4308_0 defaults
conda-package-handling 1.7.3 py38h27cfd23_1 defaults
configargparse 1.5.3 pypi_0 pypi
cryptography 3.4.7 py38hd23ed53_0 defaults
cycler 0.11.0 pypi_0 pypi
dash 2.7.0 pypi_0 pypi
dash-core-components 2.0.0 pypi_0 pypi
dash-html-components 2.0.0 pypi_0 pypi
dash-table 5.0.0 pypi_0 pypi
debugpy 1.6.0 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
descartes 1.1.0 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
executing 0.8.3 pypi_0 pypi
fastjsonschema 2.15.3 pypi_0 pypi
fire 0.4.0 pypi_0 pypi
flake8 6.0.0 pypi_0 pypi
flask 2.2.2 pypi_0 pypi
fonttools 4.33.3 pypi_0 pypi
google-auth 2.8.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
grpcio 1.46.3 pypi_0 pypi
idna 2.10 pyhd3eb1b0_0 defaults
imageio 2.22.4 pypi_0 pypi
importlib-metadata 4.11.4 pypi_0 pypi
importlib-resources 5.8.0 pypi_0 pypi
iniconfig 1.1.1 pypi_0 pypi
ipykernel 6.15.0 pypi_0 pypi
ipython 8.4.0 pypi_0 pypi
ipython-genutils 0.2.0 pypi_0 pypi
ipywidgets 7.7.0 pypi_0 pypi
itsdangerous 2.1.2 pypi_0 pypi
jedi 0.18.1 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
joblib 1.1.0 pypi_0 pypi
json5 0.9.8 pypi_0 pypi
jsonschema 4.6.0 pypi_0 pypi
jupyter 1.0.0 pypi_0 pypi
jupyter-client 7.3.4 pypi_0 pypi
jupyter-console 6.4.4 pypi_0 pypi
jupyter-core 4.10.0 pypi_0 pypi
jupyter-server 1.17.1 pypi_0 pypi
jupyterlab 3.4.3 pypi_0 pypi
jupyterlab-language-pack-zh-cn 3.4.post1 pypi_0 pypi
jupyterlab-pygments 0.2.2 pypi_0 pypi
jupyterlab-server 2.14.0 pypi_0 pypi
jupyterlab-widgets 1.1.0 pypi_0 pypi
kiwisolver 1.4.3 pypi_0 pypi
ld_impl_linux-64 2.35.1 h7274673_9 defaults
libffi 3.3 he6710b0_2 defaults
libgcc-ng 9.3.0 h5101ec6_17 defaults
libgomp 9.3.0 h5101ec6_17 defaults
libstdcxx-ng 9.3.0 hd4cf53a_17 defaults
llvmlite 0.36.0 pypi_0 pypi
lyft-dataset-sdk 0.0.8 pypi_0 pypi
markdown 3.3.7 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
matplotlib 3.5.2 pypi_0 pypi
matplotlib-inline 0.1.3 pypi_0 pypi
mccabe 0.7.0 pypi_0 pypi
mistune 0.8.4 pypi_0 pypi
mmcv 2.0.0rc3 pypi_0 pypi
mmdet 3.0.0rc4 pypi_0 pypi
mmdet3d 1.1.0rc2 dev_0 <develop>
mmengine 0.3.2 pypi_0 pypi
model-index 0.1.11 pypi_0 pypi
mypy-extensions 0.4.3 pypi_0 pypi
nbclassic 0.3.7 pypi_0 pypi
nbclient 0.6.4 pypi_0 pypi
nbconvert 6.5.0 pypi_0 pypi
nbformat 5.5.0 pypi_0 pypi
ncurses 6.2 he6710b0_1 defaults
nest-asyncio 1.5.5 pypi_0 pypi
networkx 2.8.8 pypi_0 pypi
notebook 6.4.12 pypi_0 pypi
notebook-shim 0.1.0 pypi_0 pypi
numba 0.53.0 pypi_0 pypi
numpy 1.22.4 pypi_0 pypi
nuscenes-devkit 1.1.9 pypi_0 pypi
oauthlib 3.2.0 pypi_0 pypi
open3d 0.16.0 pypi_0 pypi
opencv-python 4.6.0.66 pypi_0 pypi
openmim 0.3.3 pypi_0 pypi
openssl 1.1.1k h27cfd23_0 defaults
ordered-set 4.1.0 pypi_0 pypi
packaging 21.3 pypi_0 pypi
pandas 1.5.2 pypi_0 pypi
pandocfilters 1.5.0 pypi_0 pypi
parso 0.8.3 pypi_0 pypi
pathspec 0.10.2 pypi_0 pypi
pexpect 4.8.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 9.1.1 pypi_0 pypi
pip 21.1.3 py38h06a4308_0 defaults
platformdirs 2.5.4 pypi_0 pypi
plotly 5.11.0 pypi_0 pypi
pluggy 1.0.0 pypi_0 pypi
plyfile 0.7.4 pypi_0 pypi
prometheus-client 0.14.1 pypi_0 pypi
prompt-toolkit 3.0.29 pypi_0 pypi
protobuf 3.19.4 pypi_0 pypi
psutil 5.9.1 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
py 1.10.0 pypi_0 pypi
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pycocotools 2.0.6 pypi_0 pypi
pycodestyle 2.10.0 pypi_0 pypi
pycosat 0.6.3 py38h7b6447c_1 defaults
pycparser 2.20 py_2 defaults
pyflakes 3.0.1 pypi_0 pypi
pygments 2.12.0 pypi_0 pypi
pyopenssl 20.0.1 pyhd3eb1b0_1 defaults
pyparsing 3.0.9 pypi_0 pypi
pyquaternion 0.9.9 pypi_0 pypi
pyrsistent 0.18.1 pypi_0 pypi
pysocks 1.7.1 py38h06a4308_0 defaults
pytest 7.1.2 pypi_0 pypi
python 3.8.10 h12debd9_8 defaults
python-dateutil 2.8.2 pypi_0 pypi
pytz 2022.1 pypi_0 pypi
pywavelets 1.3.0 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
pyzmq 23.2.0 pypi_0 pypi
qtconsole 5.4.0 pypi_0 pypi
qtpy 2.3.0 pypi_0 pypi
readline 8.1 h27cfd23_0 defaults
requests 2.25.1 pyhd3eb1b0_0 defaults
requests-oauthlib 1.3.1 pypi_0 pypi
rich 12.5.1 pypi_0 pypi
rsa 4.8 pypi_0 pypi
ruamel_yaml 0.15.100 py38h27cfd23_0 defaults
scikit-image 0.19.3 pypi_0 pypi
scikit-learn 1.1.2 pypi_0 pypi
scipy 1.9.3 pypi_0 pypi
send2trash 1.8.0 pypi_0 pypi
setuptools 52.0.0 py38h06a4308_0 defaults
shapely 1.8.5.post1 pypi_0 pypi
six 1.16.0 pyhd3eb1b0_0 defaults
sniffio 1.2.0 pypi_0 pypi
soupsieve 2.3.2.post1 pypi_0 pypi
sqlite 3.36.0 hc218d9a_0 defaults
stack-data 0.3.0 pypi_0 pypi
supervisor 4.2.4 pypi_0 pypi
tabulate 0.8.10 pypi_0 pypi
tenacity 8.0.1 pypi_0 pypi
tensorboard 2.9.1 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
termcolor 2.1.1 pypi_0 pypi
terminado 0.15.0 pypi_0 pypi
terminaltables 3.1.10 pypi_0 pypi
threadpoolctl 3.1.0 pypi_0 pypi
tifffile 2022.8.12 pypi_0 pypi
tinycss2 1.1.1 pypi_0 pypi
tk 8.6.10 hbc83047_0 defaults
tomli 2.0.1 pypi_0 pypi
torch 1.11.0+cu113 pypi_0 pypi
torchvision 0.12.0+cu113 pypi_0 pypi
tornado 6.1 pypi_0 pypi
tqdm 4.61.2 pyhd3eb1b0_1 defaults
traitlets 5.3.0 pypi_0 pypi
trimesh 2.35.39 pypi_0 pypi
typing-extensions 4.2.0 pypi_0 pypi
urllib3 1.26.6 pyhd3eb1b0_1 defaults
wcwidth 0.2.5 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
websocket-client 1.3.3 pypi_0 pypi
werkzeug 2.2.2 pypi_0 pypi
wheel 0.36.2 pyhd3eb1b0_0 defaults
widgetsnbextension 3.6.0 pypi_0 pypi
xz 5.2.5 h7b6447c_0 defaults
yaml 0.2.5 h7b6447c_0 defaults
yapf 0.32.0 pypi_0 pypi
zipp 3.8.0 pypi_0 pypi
zlib 1.2.11 h7b6447c_3 defaults安装缺少的包
pip install timm
python -m pip install /root/autodl-tmp/linshi_m3dm5/M3DM_5_autodl/Pointnet2_PyTorch-master/pointnet2_ops_lib
pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
报错
(base) root@autodl-container-a3574a91f4-10f31e68:~# sudo apt install ninja-build
Reading package lists… Done
Building dependency tree
Reading state information… Done
Package ninja-build is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package ‘ninja-build’ has no installation candidate
更新sudo apt update
再次
sudo apt install ninja-build
可以了
报错
module 'open3d' has no attribute 'registration'
改open3d为更旧点的版本
conda install open3d=0.10 -c open3d-admin
可以了
修改models.py中的RGB和点云backbone的路径
例如 ## RGB backboneself.rgb_backbone = timm.create_model(model_name=rgb_backbone_name, pretrained=True,pretrained_cfg_overlay=dict(file='/root/autodl-tmp/m3dm_checkpoints/dino_vitbase8_pretrain.pth'),**kwargs)
修改main.py路径参数
例如 parser.add_argument(‘–dataset_path’, default=‘/root/autodl-tmp/m3dmpre/mvtec3d’, type=str,
help=‘Dataset store path’)
运行
python3 /root/autodl-tmp/linshi_m3dm5/M3DM_5_autodl/main.py
–method_name DINO+Point_MAE+Fusion
–use_uff
–memory_bank multiple
–rgb_backbone_name vit_base_patch8_224_dino
–xyz_backbone_name Point_MAE
Fitting random projections. Start dim处killed
怀疑内存不够，
再增加一张卡，
/root/miniconda3/lib/python3.8/site-packages/timm/models/layers/init.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
/root/miniconda3/lib/python3.8/site-packages/timm/models/_factory.py:117: UserWarning: Mapping deprecated model name vit_base_patch8_224_dino to current vit_base_patch8_224.dino.
model = create_fn(
[Fusion Block]
Extracting train features for class bagel: 100%|████████████████████████████████████████████████████████████████████| 244/244 [04:51<00:00, 1.19s/it]
Running coreset for DINO+Point_MAE+Fusion on class bagel…
Fitting random projections. Start dim = torch.Size([765184, 1152]).
DONE. Transformed dim = torch.Size([765184, 334]).
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 76517/76517 [03:01<00:00, 422.07it/s]
Fitting random projections. Start dim = torch.Size([191296, 768]).
DONE. Transformed dim = torch.Size([191296, 300]).
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 7650/7650 [00:04<00:00, 1644.75it/s]
Fitting random projections. = torch.Size([765184, 1920]).
DONE. Transformed dim = torch.Size([765184, 334]).
Extracting test features for class bagel: 100%|█████████████████████████████████████████████████████████████████████| 110/110 [02:38<00:00, 1.44s/it]
Class: bagel, DINO+Point_MAE+Fusion Image ROCAUC: 0.998, DINO+Point_MAE+Fusion Pixel ROCAUC: 0.992, DINO+Point_MAE+Fusion AU-PRO: 0.943
可以了
下面不用看，是只和自己有关的笔记。
(T)
Running coreset for DINO+Point_MAE+Fusion on class cookie…
Fitting random projections. Start dim = torch.Size([658560, 1152]).
DONE. Transformed dim = torch.Size([1152, 174]).
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 65855/65855 [00:22<00:00, 2867.81it/s]
Fitting random projections. Start dim = torch.Size([164640, 768]).
DONE. Transformed dim = torch.Size([768, 164]).
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 6584/6584 [00:02<00:00, 3129.53it/s]
Fitting random projections. Start dim = torch.Size([658560, 1920]).
DONE. Transformed dim = torch.Size([1920, 186]).
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 6584/6584 [00:02<00:00, 2974.49it/s]
Running late fusion for DINO+Point_MAE+Fusion on class cookie…: 100%
两个一起运行的内存占用情况
(base) root@autodl-container-a3574a91f4-10f31e68:~# free -m
total used free shared buff/cache available
Mem: 515594 71831 48068 27918 395694 412191
Swap: 0 0 0
(base) root@autodl-container-a3574a91f4-10f31e68:~# free -m
total used free shared buff/cache available
Mem: 515594 95903 63180 28743 356510 387375
Swap: 0 0 0
(base) root@autodl-container-a3574a91f4-10f31e68:~# free -m
total used free shared buff/cache available
Mem: 515594 73935 85130 28742 356528 409344
Swap: 0 0 0