一

{"type":"编程笔记"}


  • Home

  • Archives
  • Search

percent encoding in rfc1738 rfc2396 and rfc3986

Posted on 2018-04-21   |   In java

URI && URL

URI(统一资源标识)定义在RFC 3986中,URL(统一资源地址)是URI的一种特殊形式,提供了资源的网络位置。URL最初的标准RFC 1738已经废弃,其中定义了URL,域名,IP地址,application/x-www-form-urlencoded格式。

encode && decode URI

encode是将URI根据RFC规范定义的reserved和unreserved之外的所有字符都percent-encoding转义,避免解析时产生岐义和错误。比如,一个用户可能会输入Thyme &time=again作为 comment 变量的一部分。如果不使用encodeURIComponent对此内容进行转义,服务器得到的将是comment=Thyme%20&time=again。请注意,&符号和=符号产生了一个新的键值对,所以服务器得到两个键值对(一个键值对是comment=Thyme,另一个则是time=again),而不是一个键值对。

encode和decode一般是配对使用,比如表单 POST 提交的内容是按application/x-www-form-urlencoded编码提交到后台的,可以用 PHP 的urldecode方法解码,在 Java 中则可以用java.net.URLDecode.decode(String s, String enc)方法来解码。如果前端使用encodeURIComponent编码后 ajax POST 到后端,则需要用 PHP 中的rawurlencode解码,在 Java 中则可以用 spring 的工具类org.springframework.web.util.UriUtils.decode(String source, String encoding)解码。

application/x-www-form-urlencoded 编码规则

5.2. application/x-www-form-urlencoded serializingThe application/x-www-form-urlencoded byte serializer takes a byte sequence input and then runs these steps:Let output be the empty string.For each byte in input, depending on byte:0x20 (SP)Append U+002B (+) to output.0x2A (*)0x2D (-)0x2E (.)0x30 (0) to 0x39 (9)0x41 (A) to 0x5A (Z)0x5F (_)0x61 (a) to 0x7A (z)Append a code point whose value is byte to output.OtherwiseAppend byte, percent encoded, to output.Return output.

如上application/x-www-form-urlencoded编码规则,alphanumeric、*、-、.、_不进行编码,空格则编码成+,其他字符全部需要编码,这就是为什么 Google 搜索时空格在URL上会变成+而不是%20。

RFC 3986 中字符定义

RFC 3986                   URI Generic Syntax               January 20052.1.  Percent-Encoding   A percent-encoding mechanism is used to represent a data octet in a   component when that octet's corresponding character is outside the   allowed set or is being used as a delimiter of, or within, the   component.  A percent-encoded octet is encoded as a character   triplet, consisting of the percent character "%" followed by the two   hexadecimal digits representing that octet's numeric value.  For   example, "%20" is the percent-encoding for the binary octet   "00100000" (ABNF: %x20), which in US-ASCII corresponds to the space   character (SP).  Section 2.4 describes when percent-encoding and   decoding is applied.      pct-encoded = "%" HEXDIG HEXDIG   The uppercase hexadecimal digits 'A' through 'F' are equivalent to   the lowercase digits 'a' through 'f', respectively.  If two URIs   differ only in the case of hexadecimal digits used in percent-encoded   octets, they are equivalent.  For consistency, URI producers and   normalizers should use uppercase hexadecimal digits for all percent-   encodings.2.2.  Reserved Characters   URIs include components and subcomponents that are delimited by   characters in the "reserved" set.  These characters are called   "reserved" because they may (or may not) be defined as delimiters by   the generic syntax, by each scheme-specific syntax, or by the   implementation-specific syntax of a URI's dereferencing algorithm.   If data for a URI component would conflict with a reserved   character's purpose as a delimiter, then the conflicting data must be   percent-encoded before the URI is formed.  reserved    = gen-delims / sub-delims      gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"      sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"                  / "*" / "+" / "," / ";" / "="2.3.  Unreserved Characters   Characters that are allowed in a URI but do not have a reserved   purpose are called unreserved.  These include uppercase and lowercase   letters, decimal digits, hyphen, period, underscore, and tilde.      unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"

RFC 2396 中的部分字符定义

Javascript 中的encodeURI方法对于rfc2396中定义的reserved和unreserved这2个字符集中的字符都不会进行转义。

2.2. Reserved Characters      reserved    = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" |                    "$" | ","2.3. Unreserved Characters      unreserved  = alphanum | mark      mark        = "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"

RFC 1738 中部分字符定义

RFC 1738            Uniform Resource Locators (URL)        December 1994safe           = "$" | "-" | "_" | "." | "+"extra          = "!" | "*" | "'" | "(" | ")" | ","reserved       = ";" | "/" | "?" | ":" | "@" | "&" | "="hex            = digit | "A" | "B" | "C" | "D" | "E" | "F" |                 "a" | "b" | "c" | "d" | "e" | "f"escape         = "%" hex hexunreserved     = alpha | digit | safe | extra
Read more »

命令行神器之 mycli 篇

Posted on 2018-04-13   |   In mysql

init mycli config file

 git clone https://github.com/yuweijun/mycli.git cd mycli mkdir -p ~/.mysql/out ln -s myclirc ~/.myclirc ln -s my.cnf ~/.my.cnf ln -s my.vim ~/.my.vim

log query sql and result

 mycli -u root -p password -D database -l ~/.mysql/audit.log

按,e关闭弹出的vim窗口,按,w保存查询SQL的结果到~/.mysql/out/[0-9]{10}.log文件中,并关闭vim窗口。

Read more »

命令行神器之 ssh 篇

Posted on 2018-04-07   |   In linux

tldr ssh

Secure Shell is a protocol used to securely log onto remote systems.It can be used for logging or executing commands on a remote server.- Connect to a remote server:      ssh username@remote_host- Connect to a remote server with a specific identity (private key):      ssh -i path/to/key_file username@remote_host- Connect to a remote server using a specific port:      ssh username@remote_host -p 2222- Run a command on a remote server:      ssh remote_host command -with -flags- SSH tunneling: Dynamic port forwarding (SOCKS proxy on localhost:9999):      ssh -D 9999 -C username@remote_host- SSH tunneling: Forward a specific port (localhost:9999 to slashdot.org:80):      ssh -L 9999:slashdot.org:80 username@remote_host- Enable the option to forward the authentication information to the remote machine (see `man ssh_config` for available options):      ssh -o "ForwardAgent=yes" username@remote_host

using ssh agent

ssh-agent命令是一种用来保存公钥身份验证所使用的私钥的程序,本质上ssh-agent就是一个密钥管理器,运行ssh-agent以后,使用ssh-add将私钥交给ssh-agent保管,其他程序需要身份验证的时候可以将验证申请交给ssh-agent来完成整个认证过程。

关于ssh-agent的工作原理可以查看An Illustrated Guide to SSH Agent Forwarding这篇文章,这里简单举例子说明其作用:

假设当前机器为A,另有远程服务器B、C、D,将A的公钥发布到3台远程服务器上,在机器A开启ssh-agent,然后ssh-add当前A机器的私钥,并且在~/.ssh/config中开启转发ForwardAgent yes,这样B、C和D三台服务器之间 ssh 登录都可以通过A的这个公钥进行验证登录,而不用配置A的私钥,比如先 ssh 到B,可以从B直接 ssh 到C和D。

A 机器执行的命令

B,C和D服务器上只要开启sshd服务,并且将用户目录下面的~/.ssh目录备份删除掉。A机器上备份删除原来的~/.ssh目录后,执行以下命令:

# 创建密钥 ssh-keygen -t rsa -C "username@A"# 一般 Mac OSX 和 Linux 都已经开启`ssh-agent`服务,可以使用以下命令确认 echo "$SSH_AUTH_SOCK"# 如果没有开启 ssh-agent,则手动输入下面这个命令 eval "$(ssh-agent -s)"# 显示ssh-agent中的公钥 ssh-add -L# 如果没有显示任何公钥信息,则手动加入个人的密钥 ssh-add ~/.ssh/id_rsa# 开启转发,一定要有这个配置,不然开启了 ssh-agent 也不会转发 echo "Host *\n    ForwardAgent yes" > ~/.ssh/config# 用 ssh-copy-id 命令发布 A 机器的公钥到 B,C,D 服务器上,不要手动去创建目录和文件 ssh-copy-id username@B ssh-copy-id username@C ssh-copy-id username@D

检查本地被代理的密钥

  • -L:显示ssh-agent中的公钥
  • -l:显示ssh-agent中的密钥
 ssh-add -LThe agent has no identities.

如果提示如上,则目前没有代理任何密钥,Linux 上可以通过ssh-add ~/.ssh/id_rsa命令添加指定的密钥。Mac OSX 上启动和添加密钥的操作说明,可以参考github文档的说明:

# Note: The -K option is Apple standard version of ssh-add, which stores the passphrase in your keychain for you when you add an ssh key to the ssh-agent. ssh-add -K ~/.ssh/id_rsaPassphrase stored in keychain: /Users/username/.ssh/id_rsaIdentity added: /Users/username/.ssh/id_rsa (/Users/username/.ssh/id_rsa) ssh-add -L2048 2f:31:ba:0c:3e:b2:70:52:00:bc:85:37:a2:a7:77:ad /Users/username/.ssh/id_rsa (RSA)

ssh -t jumpbox.server ssh remote.server

 ssh -t -i ~/.ssh/jumpbox_rsa.pub username@jumpbox.server ssh username@remote.server ssh -t username@jumpbox.server ssh username@remote.server

~/.ssh/config 文件配置

上面的命令参数有点多,通过配置~/.ssh/config文件,可以简化ssh命令连接常用服务器的参数:

Host *    ForwardAgent yesHost jumpbox.server    HostName 192.168.1.11    User username    IdentityFile ~/.ssh/jumpbox_rsa.pub    ControlMaster auto    ControlPath ~/.ssh/socket/master-%l-%r@%h:%pHost remote.server1    HostName 192.168.1.2    User username    Port 22    ForwardAgent yes    ProxyCommand ssh -i ~/.ssh/jumpbox_rsa.pub username@remote.server 'nc %h %p'Host remote.server2    HostName 192.168.1.3    User username    Port 22    ForwardAgent yes    ProxyCommand ssh jumpbox.server nc %h %p

如上配置之后,就不用在命令行里每次显示的指定jumpbox.server,可以直接连接远程服务器,这里服务器的别名不需要 DNS 解析,ssh会从~/.ssh/config中获取服务器地址:

 ssh remote.server1 ssh remote.server2
Read more »

使用exec-maven-plugin运行spring boot应用

Posted on 2018-04-06   |   In java

Maven 项目中一般使用exec-maven-plugin运行项目中有main方法的类,运行目标为exec:java时常用的参数简单说明如下:

  • exec.mainClass,运行时指定main方法所在 Java 类的完全限定名
  • exec.args,给main方法的参数

exec-maven-plugin详细的使用帮助可以用如下命令查看:

 mvn exec:help -Ddetail=true
Read more »

linux 常用命令行之 dig

Posted on 2018-04-05   |   In linux

检查当前网络使用的 DNS 服务器,以及检查不同的 DNS 服务器解析域名的情况,主要使用dig和nslookup,也可以用host检查 DNS 解析情况,下面示例的命令是在 Mac OSX 电脑上运行的。

dig

dig是domain information groper的缩写,man手册上说明简单摘录如下,详细中文说明手册可参考 IBM 的文档:

dig (domain information groper) is a flexible tool for interrogating DNS name servers. It performs DNS lookups and displays the answers that are returned from the name server(s) that were queried. Most DNS administrators use dig to troubleshoot DNS problems because of its flexibility, ease of use and clarity of output. Other lookup tools tend to have less functionality than dig.

一个典型的dig调用类似:

 dig @dns-server name type

dig命令使用示例

查看当前电脑使用的 DNS 服务器地址,默认会使用/etc/resolv.conf中的配置。

 dig; <<>> DiG 9.8.3-P1 <<>>;; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34872;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 0;; QUESTION SECTION:;.                IN  NS;; ANSWER SECTION:.         6795    IN  NS  m.root-servers.net..         6795    IN  NS  h.root-servers.net..         6795    IN  NS  d.root-servers.net..         6795    IN  NS  c.root-servers.net..         6795    IN  NS  l.root-servers.net..         6795    IN  NS  i.root-servers.net..         6795    IN  NS  e.root-servers.net..         6795    IN  NS  k.root-servers.net..         6795    IN  NS  f.root-servers.net..         6795    IN  NS  j.root-servers.net..         6795    IN  NS  g.root-servers.net..         6795    IN  NS  b.root-servers.net..         6795    IN  NS  a.root-servers.net.;; Query time: 6 msec;; SERVER: 202.96.209.133#53(202.96.209.133);; WHEN: Thu Apr  5 13:28:22 2018;; MSG SIZE  rcvd: 228

本机/etc/resolv.conf文件配置的内容如下:

nameserver 202.96.209.133nameserver 8.8.8.8

所以dig命令显示的DNS服务地址是202.96.209.133。

Read more »
1…8910…99
yuweijun

yuweijun

492 posts
12 categories
RSS
GitHub Twitter
© 2021 yuweijun
Powered by Hexo
Theme - NexT.Mist.KISS