高手的存在,就是让服务10亿人的时候,你感觉只是为你一个人服务......

tcpcopy的兄弟-gor

目录
  1. 1. 简单介绍gor
  2. 2. gor安装
  3. 3. gor常用命令
    1. 3.1. 实时引流:
    2. 3.2. 流量放大、缩小
    3. 3.3. 回放
    4. 3.4. 过滤
    5. 3.5. 自定义参数

之前写了一篇tcpcopy的使用,解决了线上流量引到测试环境的问题。

最近看到一款更简单易用的流量复制工具-gor


简单介绍gor

Gor是用Golang写的一个HTTP实时流量复制工具.

github地址:https://github.com/buger/goreplay

官方描述:

GoReplay is an open-source tool for capturing and replaying live HTTP traffic into a test environment in order to continuously test your system with real data.
It can be used to increase confidence in code deployments, configuration changes and infrastructure changes.

Alt text

做的事情跟tcpcopy一样,也支持流量的方法、缩小,回放等功能。

gor只能对http协议的流量进行复制,tcp协议的流量复制还要靠tcpcopy,就算如此也解决了工作中绝大多数的应用场景。


gor安装

https://github.com/buger/goreplay/releases ,下载最新的包解压后就可以使用了。


gor常用命令

gor –help 使用方法写的比较清楚。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
 -cpuprofile string
write cpu profile to file
-debug verbose
Turn on debug output, shows all intercepted traffic. Works only when with verbose flag
-http-allow-header value
A regexp to match a specific header against. Requests with non-matching headers will be dropped:
gor --input-raw :8080 --output-http staging.com --http-allow-header api-version:^v1 (default [])
-http-allow-method value
Whitelist of HTTP methods to replay. Anything else will be dropped:
gor --input-raw :8080 --output-http staging.com --http-allow-method GET --http-allow-method OPTIONS (default [])
-http-allow-url value
A regexp to match requests against. Filter get matched against full url with domain. Anything else will be dropped:
gor --input-raw :8080 --output-http staging.com --http-allow-url ^www. (default [])
-http-disallow-header value
A regexp to match a specific header against. Requests with matching headers will be dropped:
gor --input-raw :8080 --output-http staging.com --http-disallow-header "User-Agent: Replayed by Gor" (default [])
-http-disallow-url value
A regexp to match requests against. Filter get matched against full url with domain. Anything else will be forwarded:
gor --input-raw :8080 --output-http staging.com --http-disallow-url ^www. (default [])
-http-header-limiter value
Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific header:
gor --input-raw :8080 --output-http staging.com --http-header-imiter user-id:25% (default [])
-http-original-host
Normally gor replaces the Host http header with the host supplied with --output-http. This option disables that behavior, preserving the original Host header.
-http-param-limiter value
Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific GET param:
gor --input-raw :8080 --output-http staging.com --http-param-limiter user_id:25% (default [])
-http-rewrite-url value
Rewrite the request url based on a mapping:
gor --input-raw :8080 --output-http staging.com --http-rewrite-url /v1/user/([^\/]+)/ping:/v2/user/$1/ping (default [])
-http-set-header value
Inject additional headers to http reqest:
gor --input-raw :8080 --output-http staging.com --http-set-header 'User-Agent: Gor' (default [])
-http-set-param value
Set request url param, if param already exists it will be overwritten:
gor --input-raw :8080 --output-http staging.com --http-set-param api_key=1 (default [])
-input-dummy value
Used for testing outputs. Emits 'Get /' request every 1s (default [])
-input-file value
Read requests from file:
gor --input-file ./requests.gor --output-http staging.com (default [])
-input-http value
Read requests from HTTP, should be explicitly sent from your application:
# Listen for http on 9000
gor --input-http :9000 --output-http staging.com (default [])
-input-raw value
Capture traffic from given port (use RAW sockets and require *sudo* access):
# Capture traffic from 8080 port
gor --input-raw :8080 --output-http staging.com (default [])
-input-raw-engine libpcap
Intercept traffic using libpcap (default), and `raw_socket` (default "libpcap")
-input-tcp value
Used for internal communication between Gor instances. Example:
# Receive requests from other Gor instances on 28020 port, and redirect output to staging
gor --input-tcp :28020 --output-http staging.com (default [])
-memprofile string
write memory profile to this file
-middleware string
Used for modifying traffic using external command
-output-dummy value
Used for testing inputs. Just prints data coming from inputs. (default [])
-output-file value
Write incoming requests to file:
gor --input-raw :80 --output-file ./requests.gor (default [])
-output-http value
Forwards incoming requests to given http address.
# Redirect all incoming requests to staging.com address
gor --input-raw :80 --output-http http://staging.com (default [])
-output-http-debug
Enables http debug output.
-output-http-elasticsearch string
Send request and response stats to ElasticSearch:
gor --input-raw :8080 --output-http staging.com --output-http-elasticsearch 'es_host:api_port/index_name'
-output-http-header --output-http-header
WARNING: --output-http-header DEPRECATED, use `--http-set-header` instead (default [])
-output-http-header-filter --output-http-header-filter
WARNING: --output-http-header-filter DEPRECATED, use `--http-allow-header` instead (default [])
-output-http-header-hash-filter output-http-header-hash-filter
WARNING: output-http-header-hash-filter DEPRECATED, use `--http-header-hash-limiter` instead (default [])
-output-http-method --output-http-method
WARNING: --output-http-method DEPRECATED, use `--http-allow-method` instead (default [])
-output-http-redirects int
Enable how often redirects should be followed.
-output-http-rewrite-url --output-http-rewrite-url
WARNING: --output-http-rewrite-url DEPRECATED, use `--http-rewrite-url` instead (default [])
-output-http-stats
Report http output queue stats to console every 5 seconds.
-output-http-timeout duration
Specify HTTP request/response timeout. By default 5s. Example: --output-http-timeout 30s
-output-http-url-regexp --output-http-url-regexp
WARNING: --output-http-url-regexp DEPRECATED, use `--http-allow-url` instead (default [])
-output-http-workers int
Gor uses dynamic worker scaling by default. Enter a number to run a set number of workers.
-output-tcp value
Used for internal communication between Gor instances. Example:
# Listen for requests on 80 port and forward them to other Gor instance on 28020 port
gor --input-raw :80 --output-tcp replay.local:28020 (default [])
-output-tcp-stats
Report TCP output queue stats to console every 5 seconds.
-split-output true
By default each output gets same traffic. If set to true it splits traffic equally among all outputs.
-stats
Turn on queue stats output
-verbose
Turn on more verbose output

常用的命令:

线上引流的机器为192.168.1.1 ,测试机器为192.168.2.2

实时引流:

gor –input-raw :80 –output-http “192.168.2.2:80”

把192.168.1.1 80端口上的http请求转到192.168.2.2 80端口上。

gor –input-tcp :80–output-http “192.168.2.2:80” –output-http “192.168.3.3:80”

把流量复制两份到不同的测试机器


流量放大、缩小

gor –input-raw :80 –output-http “192.168.2.2:80|1000%”

流量放大10倍

gor –input-raw :80 –output-http “192.168.2.2:80|10%”

流量缩小10倍


回放

gor –input-raw :80–output-file requests.gor

保存请求到文件中

gor –input-file requests.gor –output-http “192.168.2.2:80”

回放请求


过滤

gor –input-raw :80–output-http 192.168.2.2:80 –output-http-url-regexp ^www.

只保留www开头的请求


自定义参数

gor –input-raw :80–output-http 192.168.2.2:80 –http-set-header ‘User-Agent: gor’

在请求头上加入 User-Agent: gor


关于gor的其他特性,详见官方wiki:
https://github.com/buger/goreplay/wiki