【Go 开发】pprof 排查问题流程:排查程序 CPU 占用高的问题
原理部分
- 引入 net/http/pprof 进行代码中的分析
import _ "net/http/pprof"
import "net/http"func main() {// 启动 pprof 服务器go func() {http.ListenAndServe("localhost:6060", nil)}()// 其他业务代码
}
这样会在 localhost:6060 启动一个 HTTP 服务器,可以在浏览器打开 http://localhost:6060/debug/pprof/ 查看各类 pprof 信息。
- 生成和查看 pprof 文件
在终端使用 go tool pprof 命令生成和查看 pprof 文件。例如:
go tool pprof http://localhost:6060/debug/pprof/heap # 查看内存使用情况
go tool pprof http://localhost:6060/debug/pprof/profile # 查看 CPU 使用情况
你还可以将这些信息保存为文件以供离线分析:
curl -o heap.prof http://localhost:6060/debug/pprof/heap
curl -o cpu.prof http://localhost:6060/debug/pprof/profile?seconds=30
go tool pprof heap.prof
go tool pprof cpu.prof
-
常见的 pprof 分析命令
进入 pprof 交互式命令行界面后,可以使用以下命令进行分析:
• top:显示 CPU 或内存消耗最多的函数,默认按消耗排序。
• list <function_name>:查看指定函数的源码以及性能消耗信息。
• web:以图形界面展示性能分析结果(需要安装 Graphviz)。
• svg:生成 SVG 格式的分析图。
• png:生成 PNG 格式的分析图。 -
分析 CPU 和内存的使用情况
在排查问题时,可以重点关注以下两方面:
• CPU Profiler:用于分析 CPU 使用情况,帮助找到耗时较长的代码段。
• Heap Profiler:用于分析内存使用情况,帮助找到内存泄漏和过高的内存使用问题。 -
常见性能瓶颈
- CPU 使用过高:检查是否有热点函数(CPU 消耗较高的函数),并尝试优化相关代码。
- 内存泄漏:关注 heap 的 top 输出中是否有内存增长,或者某些数据结构未释放。
- 阻塞操作:查看 goroutine 的 profile,检查是否存在锁竞争或通道阻塞问题。
实例演示
内存
go tool pprof http://localhost:6060/debug/pprof/heap
Fetching profile over HTTP from http://localhost:6060/debug/pprof/heap
Saved profile in /home/hope/pprof/pprof.td2_data_center_new.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
File: td2_data_center_new
Build ID: 25244974efa19a16891d82f43c0aca01f0ccd04b
Type: inuse_space
Time: Nov 13, 2024 at 3:53pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 510.56MB, 94.54% of 540.07MB total
Dropped 34 nodes (cum <= 2.70MB)
Showing top 10 nodes out of 54flat flat% sum% cum cum%244.62MB 45.29% 45.29% 244.62MB 45.29% td2_data_center/app.LoadPushConf185.02MB 34.26% 79.55% 185.52MB 34.35% gitlab.yeahka.com/tdcode2_go/pkg/river.(*TableFilter).slice2Map22.90MB 4.24% 83.79% 22.90MB 4.24% github.com/confluentinc/confluent-kafka-go/kafka.NewProducer20.52MB 3.80% 87.59% 20.52MB 3.80% github.com/go-mysql-org/go-mysql/packet.(*Conn).ReadPacketReuseMem12MB 2.22% 89.81% 12MB 2.22% bufio.NewWriterSize9MB 1.67% 91.48% 19.50MB 3.61% github.com/go-mysql-org/go-mysql/replication.(*RowsEvent).decodeValue4.50MB 0.83% 92.31% 4.50MB 0.83% time.Time.Format4MB 0.74% 93.06% 4MB 0.74% encoding/json.Marshal4MB 0.74% 93.80% 4MB 0.74% github.com/syndtr/goleveldb/leveldb/memdb.New4MB 0.74% 94.54% 155.42MB 28.78% gitlab.yeahka.com/tdcode2_go/pkg/river.(*TableFilter).parseUpdate
(pprof) list app.LoadPushConf
Total: 540.07MB
ROUTINE ======================== td2_data_center/app.LoadPushConf in /home/hope/go/src/td2_data_center/app/config.go244.62MB 244.62MB (flat, cum) 45.29% of Total. . 205: for _, pushItem := range MyPushInfo.PushConfig {. . 206: // 每个商户都对应一个chan. . 207: PushChanMapTmp := make(map[int]chan *river.RowData). . 208: for i := 0; i < MyCfg.PushCoMax; i++ {. . 209: //pushChan := make(chan *river.RowData, 8000)244.62MB 244.62MB 210: pushChan := make(chan *river.RowData, 10000). . 211: PushChanMapTmp[i] = pushChan. . 212: }. . 213:. . 214: // pushChan 作为推送chan的标识,订单和流水这类有时序性的需要放在一起. . 215: _, ok := PushChanMapList[pushItem.TableFilter.PushChan]
(pprof) svg
Generating report in profile002.svg
CPU
hope@apptest42 td2_data_center]$ go tool pprof http://localhost:6060/debug/pprof/heap
Fetching profile over HTTP from http://localhost:6060/debug/pprof/heap
Saved profile in /home/hope/pprof/pprof.td2_data_center_new.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz
File: td2_data_center_new
Build ID: 25244974efa19a16891d82f43c0aca01f0ccd04b
Type: inuse_space
Time: Nov 13, 2024 at 3:55pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 510.56MB, 94.54% of 540.07MB total
Dropped 34 nodes (cum <= 2.70MB)
Showing top 10 nodes out of 54flat flat% sum% cum cum%244.62MB 45.29% 45.29% 244.62MB 45.29% td2_data_center/app.LoadPushConf185.02MB 34.26% 79.55% 185.52MB 34.35% gitlab.yeahka.com/tdcode2_go/pkg/river.(*TableFilter).slice2Map22.90MB 4.24% 83.79% 22.90MB 4.24% github.com/confluentinc/confluent-kafka-go/kafka.NewProducer20.52MB 3.80% 87.59% 20.52MB 3.80% github.com/go-mysql-org/go-mysql/packet.(*Conn).ReadPacketReuseMem12MB 2.22% 89.81% 12MB 2.22% bufio.NewWriterSize9MB 1.67% 91.48% 19.50MB 3.61% github.com/go-mysql-org/go-mysql/replication.(*RowsEvent).decodeValue4.50MB 0.83% 92.31% 4.50MB 0.83% time.Time.Format4MB 0.74% 93.06% 4MB 0.74% encoding/json.Marshal4MB 0.74% 93.80% 4MB 0.74% github.com/syndtr/goleveldb/leveldb/memdb.New4MB 0.74% 94.54% 155.42MB 28.78% gitlab.yeahka.com/tdcode2_go/pkg/river.(*TableFilter).parseUpdate
(pprof) list app.LoadPushConf
Total: 540.07MB
ROUTINE ======================== td2_data_center/app.LoadPushConf in /home/hope/go/src/td2_data_center/app/config.go244.62MB 244.62MB (flat, cum) 45.29% of Total. . 205: for _, pushItem := range MyPushInfo.PushConfig {. . 206: // 每个商户都对应一个chan. . 207: PushChanMapTmp := make(map[int]chan *river.RowData). . 208: for i := 0; i < MyCfg.PushCoMax; i++ {. . 209: //pushChan := make(chan *river.RowData, 8000)244.62MB 244.62MB 210: pushChan := make(chan *river.RowData, 10000). . 211: PushChanMapTmp[i] = pushChan. . 212: }. . 213:. . 214: // pushChan 作为推送chan的标识,订单和流水这类有时序性的需要放在一起. . 215: _, ok := PushChanMapList[pushItem.TableFilter.PushChan]
(pprof) web
exec: "sensible-browser": executable file not found in $PATH
(pprof) svg
Generating report in profile001.svg