Socket Types
- Stream Socket SOCK_STREAM 串流式
- Datagram Socket SOCK_DGRAM 訊息式
- Row Socket
- Link Layer Socket
基於TCP
資料需要可靠的、需要順序的多半用stream socket
例如：HTTP/TELNET
基於UDP
不可靠、不一定照順序、但是如果送達(封包重組完成，如果有分割的話)，資料就是正確的。
遊戲玩家位置、音樂、音效，等可容許小量丟失的。
例如：TFT/DHCPD
2019年2月15日 星期五
2019年2月13日 星期三
RabbitMQ
基本概念
官方網頁RabbitMQ就像郵局，唯一不同的是它送數位資料，郵局送信。
Queue就是郵筒，存在郵局裡(RabbitMQ)，只受限於ram跟disk。
message只存在queue裡。
多個producers可以送message給同一個queue。
多個consumer可以從同一個queue收message。
基本上每個message被收走就沒有了。
- RabbitMQ=郵局
- Queue=郵筒(存在於郵局RabbitMQ)
- Message=信
- Exchange=在producer這邊，他是收信的工具、在consumer這邊，他是送信的工具(存在於郵局RabbitMQ)
- Producer=寄信者
- Consumer=收信者
連線架構
TCP connection -> connection -> channel
Message 送出後，consumer會回傳ack(acknowledgement)，
確認message正確收取，否則會re-queue此message。
訊息模型
Producer -> exchange -> Queue -> exchange -> Consumer
Queue -> exchange -> Consumer
以下為完成傳遞必要項目，缺一不可
- 1 由Producer產生Message
- 2 Message送到左邊的Exchange
- 3 左邊的Exchange把Message送到綁定的Queue
- 4 右邊的Exchange從Queue取出Message
- 5 右邊的Exchange把Message送給Consumer
Producer及Consumer都可以宣告Exchange及Queue
若是沒宣告就是使用預設的，或是RabbitMQ會自行建立
宣告同名的Exchange或是Queue，就表示用現有的，RabbitMQ不會新建立一個
宣告同名的Exchange或是Queue，參數必須完全相同，否則報錯
基本模型(Producer/Consumer)
Producer -> Queue -> Consumer
Producer/Consumer是位於app，Queue是位於RabbitMQ
1 massage平均分配給多個consumer
2 等consumer回覆之後，才送新的message下去
所有要收的consumer，自行綁定上該exchange
一 Produce/Consumer(基本模型)
官方網頁二 Work Queues(message acknowledgement、message durable)
官方網頁- Round-robin dispatching 輪詢式派送
輪流派送給每個註冊的worker
- Massage acknowledgement 訊息確認
msgs, err := ch.Consume( q.Name, // queue "", // consumer false, // auto-ack false, // exclusive false, // no-local false, // no-wait nil, // args ) //幾本上auto-ack都要設成true，不然就要自己送 //否則rabbitmq 會保留此message，造成資源消耗 go func() { for d := range msgs { log.Printf("Received a message: %s", d.Body) dot_count := bytes.Count(d.Body, []byte(".")) t := time.Duration(dot_count) time.Sleep(t * time.Second) log.Printf("Done") d.Ack(false) } }() - Massage durability 訊息耐用(即便rabbitmq重開，未送出的message也不會消失)
q, err := ch.QueueDeclare( "hello", // name true, // durable false, // delete when unused false, // exclusive false, // no-wait nil, // arguments ) //producer、consumer宣告的queue durable要設成true //宣告同名的queue參數必須相同，否則錯誤 err = ch.Publish( "", // exchange q.Name, // routing key false, // mandatory false, amqp.Publishing { DeliveryMode: amqp.Persistent, ContentType: "text/plain", Body: []byte(body), }) //publishing DeliveryMode 要設成 amqp.Persistent - 完成一個work才繼續派送(Fair dispatch 一節)
基本上RabbitMQ的派送是平均派送，當他收到就派送， 所以有可能派送給還沒完成工作的worker。 可設定prefetch count=1 這時rabbitMQ會收到前一個message的acknowledgement才會派送下一個。 注意，此時queue有可能被填滿，你需要多設定幾個woker。 err = ch.Qos( 1, // prefetch count 0, // prefetch size false, // global )
三 Publish/Subscribe(one message to multiple consumers)
官方網頁Published message are going to be broadcast to all the receivers.
- Exchanges
4 types of exchange:direct,topic,headers,fanout err = ch.ExchangeDeclare( "logs", // name "fanout", // type true, // durable false, // auto-deleted false, // internal false, // no-wait nil, // arguments ) //fanout:broadcast all messages to all the queues it knows
- Bindings(Binding exchange and queue)
err = ch.QueueBind( q.Name, // queue name "", // routing key "logs", // exchange false, nil )
四 Routing(用路由方式傳送message給不同Queue)
官方網頁Subscribe only to a subset of the messages.
- 基本概念
exchange publish時，可以加上路由(routing key) queue可以綁定exchange及routing key 一個queue可以綁定一個exchange及多個routing key 這樣就可以一個exchange 用多種routing發布，然後consumer利用不同routing接收
- Bindings
//Binding 是綁定exchange 及 queue //Binding時可以設定路由，用routing key parameter err = ch.QueueBind( q.Name, // queue name "black", // routing key "logs", // exchange false, nil) //route to black
- Direct exchange
//用direct exchange才能使用binding的routing key //用fanout exchange會忽略routing key參數，合理，所以他才叫fanout err = ch.ExchangeDeclare( "logs_direct", // name "direct", // type true, // durable false, // auto-deleted false, // internal false, // no-wait nil, // arguments ) q, err := ch.QueueDeclare( "", // name false, // durable false, // delete when usused true, // exclusive false, // no-wait nil, // arguments ) err = ch.QueueBind( q.Name, // queue name "", // routing key "logs", // exchange false, nil )
五 Publish/Subscribe(one message to multiple consumers)
官方網頁Published message are going to be broadcast to all the receivers.
- Exchanges
4 types of exchange:direct,topic,headers,fanout err = ch.ExchangeDeclare( "logs", // name "fanout", // type true, // durable false, // auto-deleted false, // internal false, // no-wait nil, // arguments ) //fanout:broadcast all messages to all the queues it knows
- Bindings(Binding exchange and queue)
err = ch.QueueBind( q.Name, // queue name "", // routing key "logs", // exchange false, nil )
正式產品需要注意的主題
官方說明文件- Connection Management
- Error Handling
- Connection Recovery
- Concurrency
- Metric Collection
Publisher Confirms and Consumer Acknowledgements
Production Checklist
Mornitoring
linux安裝
主目錄 /opt/rabbitmq看相關變數 printenv | grep rabbitmq
執行檔 /opt/rabbitmq/sbin
執行檔 /usr/local/bin
服務啟動
systemctl start rabbitmq-server
mac home brew 安裝
官網說明
相關的scripts及cli tools安裝在/usr/local/opt/rabbitmq/sbin，
需要自己加到path
export PATH=$PATH:/usr/local/opt/rabbitmq/sbin
brew services start rabbitmq brew services stoprabbitmq啟動在前景
/usr/local/Cellar/rabbitmq/3.7.11/sbin/rabbitmq-server
管理 rabbitmqctl
用這個程式管理rabbitmq大部分項目，像是增加使用者、列出使用者、重啟、列出參數等。範例
增加使用者 add_user [userName][password]
rabbitmqctl add_user admin admin更改使用者tag，改成管理者 set_user_tags
rabbitmqctl set_user_tags admin administrator更改使用者權限 set_permissions [name]
rabbitmqctl set_permissions -p / username ".*" ".*" ".*"更改密碼 change_password [username] [password]
rabbitmqctl change_password admin admin查詢未確認的message(unacknowledged) list_queues [queueName] messages_ready messages_unacknowledged
sudo rabbitmqctl list_queues queueName messages_ready messages_unacknowledged
監控
監控程式啟動在 http://localhost:15672 使用瀏覽器就可以開啟，要先建立一個userrabbitmq-plugins enable rabbitmq-management
外掛管理
rabbitmq-plugins [-n [node]] [-t [timeout]] [-l] [-q] [command] [command options] Eg. rabbitmq-plugins [-n] [-t ] [-l] [-q] is_enabled [plugin1] [plugin2]
參數
-n node -q quiet -h help list 列出所有外掛 is_enabled [plugin1][,[plugin2]] 查看plugin是否啟用 enable [plugin] 啟用外掛
範例
啟用外掛
rabbitmq-plugins enable rabbitmq_management
管理工具
rabbitmqadmin2019年2月5日 星期二
Docker commands
build
官方說明：用來建立image
常用參數
-t 指定repository:tag docker build . -t repository:tag eg: docker build . it myname/golang:1.14-centos7 -f 指定dockerfile docker build -f dockerfile.debug build with RUL docker build github.com/create/docker-firefox
run
官方說明：用來執行docker
使用概念
- 1 參數
- 2 綁定
- 3 執行的image
常用參數
--interactive,-i Keep STDIN open even if not attached --tty,-t Allocate a pseudo-TTY --ulimit Ulimit options --mount Attach a filesystem mount to the container --add-host Add a custom host-to-IP mapping(host:ip) --attach,-a Attach to STDIN,STDOUT or STDERR --link Add link to another container --name Assign a name to the container --publish,-p Publish a container's port(s) to the host --publish-all, -P Publish all exposed ports to random ports --expose expose a port --rm Automatically remove the container when it exits --ip IPv4 Address --ip6 IPv6 address -w lets the container command being executed inside directory given
Example
Assign name and allocate pseudo-TTY$ docker run --name test -it debianSet working directory
$ docker run -w /path/to/dir/ -i -t ubuntu pwdExporse port 80 of the container without publishing the port to host
Expose port但是沒有連結到host的外面。
$ docker run --expose 80 ubuntu bashPublish a port
把container port 發布到host介面。
這裡是綁定8080到 host tcp 80 on 127.0.0.1
$ docker run -p 127.0.0.1:80:8080/tcp ubuntu bash
2019年1月24日 星期四
Linux command netstat
netstat
netstat常用說明：用來顯示網路系統的資訊(連線、路由表、網路卡等等)
使用概念
- 第一個參數，決定資訊類型選擇
- 輸出格式控制(format)
Type of information
(none) By default, netstat displays a list of open sockets.
If you don't specify any address families,
then the active sockets of all configured address families will be printed.
--route , -r
Display the kernel routing tables. See the description in route(8) for details.
netstat -r and route -e produce the same output.
--groups , -g
Display multicast group membership information for IPv4 and IPv6.
--interfaces=iface , -I=iface , -i
Display a table of all network interfaces, or the specified iface.
--masquerade , -M
Display a list of masqueraded connections.
--statistics , -s(統計)
Display summary statistics for each protocol.
None Options(none時候的參數)
netstat [address_family_options]
[--tcp|-t] [--udp|-u] [--udplite|-U] [--sctp|-S] [--raw|-w]
[--listening|-l]
[--all|-a]
[--numeric|-n] [--numeric-hosts] [--numeric-ports] [--numeric-users]
[--symbolic|-N]
[--extend|-e[--extend|-e]]
[--timers|-o] [--program|-p]
[--verbose|-v]
[--continuous|-c]
[--wide|-W]
[delay]
Common Options
--verbose , -v
Tell the user what is going on by being verbose.
Especially print some useful information about unconfigured address families.
--wide , -W
Do not truncate IP addresses by using output as wide as needed.
This is optional for now to not break existing scripts.
--numeric , -n
Show numerical addresses instead of trying to determine symbolic host,
port or user names.
--numeric-hosts
shows numerical host addresses but does not affect the resolution
of port or user names.
--numeric-ports
shows numerical port numbers but does not affect the resolution
of host or user names.
--numeric-users
shows numerical user IDs but does not affect the resolution of host or port names.
--protocol=family , -A
Specifies the address families (perhaps better described as low level protocols)
for which connections are to be shown. family is a comma (',') separated list of
address family keywords like inet, inet6, unix, ipx, ax25, netrom, econet, and ddp.
This has the same effect as using the --inet|-4, --inet6|-6, --unix|-x, --ipx,
--ax25, --netrom, and --ddp options.
The address family inet (Iv4) includes raw, udp, udplite and tcp protocol sockets.
-c, --continuous
This will cause netstat to print the selected information every second continuously.
-e, --extend
Display additional information. Use this option twice for maximum detail.
-o, --timers
Include information related to networking timers.
-p, --program
Show the PID and name of the program to which each socket belongs.
-l, --listening
Show only listening sockets. (These are omitted by default.)
-a, --all
Show both listening and non-listening
(for TCP this means established connections) sockets.
With the --interfaces option, show interfaces that are not up
-F
Print routing information from the FIB. (This is the default.)
-C
Print routing information from the route cache.
delay
Netstat will cycle printing through statistics every delay seconds.
Output(重要的輸出項目)
Proto
The protocol (tcp, udp, udpl, raw) used by the socket.
Recv-Q
Established: The count of bytes not copied by the user program connected to this socket.
Listening: Since Kernel 2.6.18 this column contains the current syn back‐log.
Send-Q
Established: The count of bytes not acknowledged by the remote host.
Listening: Since Kernel 2.6.18 this column contains the maximum size of the syn backlog.
Local Address
Address and port number of the local end of the socket.
Unless the --numeric (-n) option is specified,
the socket address is resolved to its canonical host name
(FQDN), and the port number is translated into the corresponding service name.
Foreign Address
Address and port number of the remote end of the socket. Analogous to "Local Address."
State
The state of the socket. Since there are no states in raw mode
and usually no states used in UDP and UDPLite, this column may be left blank.
Normally this can be one of several values:
ESTABLISHED
The socket has an established connection.
SYN_SENT
The socket is actively attempting to establish a connection.
SYN_RECV
A connection request has been received from the network.
FIN_WAIT1
The socket is closed, and the connection is shutting down.
FIN_WAIT2
Connection is closed, and the socket is waiting for a shutdown from the remote end.
TIME_WAIT
The socket is waiting after close to handle packets still in the network.
CLOSE The socket is not being used.
CLOSE_WAIT
The remote end has shut down, waiting for the socket to close.
LAST_ACK
The remote end has shut down, and the socket is closed. Waiting for acknowledgement.
LISTEN The socket is listening for incoming connections.
Such sockets are not included in the output
unless you specify the --listening (-l) or --all (-a) option.
CLOSING
Both sockets are shut down but we still don't have all our data sent.
UNKNOWN
The state of the socket is unknown.
User
The username or the user id (UID) of the owner of the socket.
PID/Program name
Slash-separated pair of the process id (PID) and process name of the process that owns the socket.
--program causes this column to be included. You will also
need superuser privileges to see this information on sockets you don't own.
This identification information is not yet available for IPX sockets.
Example
顯示所有listening or established的tcpnetstat -at
顯示所有listening的tcp
netstat -lt
顯示tcp的統計
netstat -st
顯示每個tcp socket所屬的程式PID跟名稱
netstat -apt //listening and established netstat -pt //established only netstat -ap | grep ssh //使用grep篩選
Linux command ulimit
ulimit
說明：顯示/設定使用者、群組等級的shell、process使用的資源限制
使用概念
- 針對本session shell或session shell中的process做設置
- 直接顯示或設定資源限制
- 設定分為soft and hard
- 使用ulimit做的設定，都是暫時性的(只針對本session)
- 全系統的永久性修改，須改 /etc/security/limits.conf
Options
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
-H: hard limit 執行緒不能在執行中改變數值(除非有root權限)
-S: soft limit 執行緒可在執行中改變數值
不加 -H,-S，默认兩者都设置。
LIMIT 的值，除了可以是数字，也可以是
hard， soft， unlimited
hard: 当前 hard 限制值
soft: 当前 soft 限制值
unlimited: 无限制
不加 LIMIT，表示打印对应选项的资源（有多个选项时，会显示资源名和单位）：
#ulimit -f
unlimited
#ulimit -c
0
#ulimit -c -f
core file size (blocks, -c) 0
file size (blocks, -f) unlimited
-a All current limits are reported
-b The maximum socket buffer size
-c The maximum size of core files created
-d The maximum size of a process's data segment
-e The maximum scheduling priority ("nice")
-f The maximum size of files written by the shell and its
children
-i The maximum number of pending signals
-l The maximum size that may be locked into memory
-m The maximum resident set size (many systems do not honor
this limit)
-n The maximum number of open file descriptors (most systems
do not allow this value to be set)
-p The pipe size in 512-byte blocks (this may not be set)
-q The maximum number of bytes in POSIX message queues
-r The maximum real-time scheduling priority
-s The maximum stack size
-t The maximum amount of cpu time in seconds
-u The maximum number of processes available to a single user
-v The maximum amount of virtual memory available to the
shell and, on some systems, to its children
-x The maximum number of file locks
-T The maximum number of threads
If limit is given, and the -a option is not used, limit is the
new value of the specified resource. If no option is given,
then -f is assumed. Values are in 1024-byte increments, except
for -t, which is in seconds; -p, which is in units of 512-byte
blocks; and -T, -b, -n, and -u, which are unscaled values. The
return status is 0 unless an invalid option or argument is sup‐
plied, or an error occurs while setting a new limit.
Example
顯示當前user 的open files(n)的hard設定ulimit -Hn 1024顯示當前user 所有設定
ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 31204 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 (file descriptor) pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 31204 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
把當前user的file descriptor數量暫時修改為10240
#ulimit -n 10240
MySQL 維運
查詢連線數
mysqladmin -u root -p -h127.0.0.1 status
Threads 就是連線數 Uptime: 168765 Threads: 3 Questions: 2265 Slow queries: 0 Opens: 394 Flush tables: 2 Open tables: 250 Queries per second avg: 0.013
mysql>show full processlist
+----+-----------------+-----------------+------+---------+--------+------------------------+-----------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+-----------------+-----------------+------+---------+--------+------------------------+-----------------------+ | 4 | event_scheduler | localhost | NULL | Daemon | 169192 | Waiting on empty queue | NULL | | 16 | root | localhost:56512 | live | Sleep | 23923 | | NULL | | 18 | root | localhost | NULL | Query | 0 | starting | show full processlist | +----+-----------------+-----------------+------+---------+--------+------------------------+-----------------------+
2019年1月23日 星期三
Linux command ps
ps(process status)
說明：用來顯示執行緒的資訊
支援多種options：UNIX options
BSD options
GNU long options
使用概念
- 選擇process，simple selection / selection by list
- 輸出格式控制(format)
Simple selection 詳細參考man ps
a Lift the BSD-style "only yourself" restriction,
which is imposed upon the set of all processes when some BSD-style (without "-")
options are used or when the ps personality setting is BSD-like.
The set of processes selected in this manner is in addition to the set of processes selected by other means.
An alternate description is that this option causes ps to list all processes with a terminal (tty),
or to list all processes when used together with the x option.
-A Select all processes. Identical to -e.
-a Select all processes except both session leaders (see getsid(2)) and processes not associated with a terminal.
-d Select all processes except session leaders.
--deselect
Select all processes except those that fulfill the specified conditions (negates the selection).
Identical to -N.
-e Select all processes. Identical to -A.
g Really all, even session leaders. This flag is obsolete and may be discontinued in a future release.
It is normally implied by the a flag, and is only
useful when operating in the sunos4 personality.
-N Select all processes except those that fulfill the specified conditions (negates the selection).
Identical to --deselect.
T Select all processes associated with this terminal. Identical to the t option without any argument.
r Restrict the selection to only running processes.
x Lift the BSD-style "must have a tty" restriction, which is imposed upon the set of all processes
when some BSD-style (without "-") options are used or
when the ps personality setting is BSD-like. The set of processes selected in this manner is in addition
to the set of processes selected by other means.
An alternate description is that this option causes ps to list all processes owned by you (same EUID as ps),
or to list all processes when used together
with the a option.
Selection by list
ps -p "1 2" -p 3,4
-123 Identical to --pid 123.
123 Identical to --pid 123.
-C cmdlist
Select by command name. This selects the processes whose executable name is given in cmdlist.
-g grplist
Select by session OR by effective group name. Selection by session is specified by many standards,
but selection by effective group is the logical behavior that several other operating systems use.
This ps will select by session when the list is completely numeric (as sessions are). Group ID
numbers will work only when some group names are also specified. See the -s and --group options.
p pidlist
Select by process ID. Identical to -p and --pid.
--sid sesslist
Select by session ID. Identical to -s.
-u userlist
Select by effective user ID (EUID) or name. This selects the processes
whose effective user name or ID is in userlist.
Output control
-f Do full-format listing.
This option can be combined with many other UNIX-style options to add additional columns.
It also causes the command arguments to be printed.
When used with -L, the NLWP (number of threads) and LWP (thread ID) columns will be added.
See the c option, the format keyword args, and the format keyword comm.
-F Extra full format.
-j Jobs format
-o format List user-defined format.
EXAMPLES
顯示使用者當次登入資訊[root@li1548-65 live]# ps PID TTY TIME CMD 24539 pts/1 00:00:00 bash 24761 pts/1 00:00:00 ps
列出最耗費記憶體的執行緒
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head
-e：輸出所有行程
-o：指定輸出欄位，後面接著所有想要輸出的欄位名稱：
pid：行程 ID（process ID）
ppid：父行程 ID（parent process ID）
cmd：程式名稱
%mem：記憶體使用量（百分比）
%cpu：CPU 使用量（百分比）
--sort 參數則是指定排序的依據欄位，
預設會依照數值由小到大排序，
若要由大到小則在欄位名稱前加負號。
-%mem，就是記憶體使用量從大到小排序。
PID PPID CMD %MEM %CPU 2124 1278 /usr/lib/chromium-browser/c 25.0 14.4 1446 1278 /usr/lib/chromium-browser/c 21.5 15.4 1253 1 /usr/lib/chromium-browser/c 19.2 13.0 1328 1278 /usr/lib/chromium-browser/c 7.6 2.9 1392 1278 /usr/lib/chromium-browser/c 7.6 0.5 732 669 /usr/bin/X :0 -seat seat0 - 5.7 2.0 1060 1 /usr/lib/arm-linux-gnueabih 1.8 0.0 1086 758 pcmanfm --desktop --profile 1.5 0.2 1085 758 lxpanel --profile LXDE-pi 1.5 0.3
輸出pid,ppid,cpu,mem,起始時間....
ps -eo uname,pid,ppid,nlwp,pcpu,pmem,psr,start_time,tty,time,args USER PID PPID NLWP %CPU %MEM PSR START TT TIME COMMAND root 1 0 1 0.0 0.1 1 Feb20 ? 00:00:01 /sbin/init root 2 0 1 0.0 0.0 0 Feb20 ? 00:00:00 [kthreadd] root 3 2 1 0.0 0.0 0 Feb20 ? 00:02:23 [ksoftirqd/0] root 6 2 1 0.0 0.0 0 Feb20 ? 00:00:00 [migration/0] root 7 2 1 0.0 0.0 1 Feb20 ? 00:00:00 [migration/1] root 9 2 1 0.1 0.0 1 Feb20 ? 00:13:52 [ksoftirqd/1]
列出記憶體使用
ps aux | grep apache2 | awk '{ total += $6; } END { print total/1024"MB" }'
//寫成function
$ vim .bashrc
function memusage() {
ps aux | grep "$1" | awk '{ total += $6; } END { print total/1024"MB" }'
}
//用法
memusage apache2 //就可以印出記憶體使用量
列出CPU使用
ps aux | grep apache2 | awk '{ total += $3; } END { print total"%" }'
//寫成function
$ vim .bashrc
function cpuusage() {
ps aux | grep "$1" | awk '{ total += $3; } END { print total"%" }'
}
cpuusage apache2 # 就可以印出總 CPU 使用量
註：cpu 算法是比較奇怪(因為包含多顆 CPU)，不過可以把自己的 CPU 數量 * 100%，再來看這個比例。
訂閱：
文章 (Atom)