December 18, 2013

The best grep replacement: ag

It's syntax is pretty much the same as ack, but written in optimized C and is noticeably faster.

December 6, 2013

The Telnet Protocol

The Telnet protocol is often thought of as simply providing a facility for remote logins to computer via the Internet. This was its original purpose although it can be used for many other purposes. It is best understood in the context of a user with a simple terminal using the local telnet program (known as the client program) to run a login session on a remote computer where his communications needs are handled by a telnet server program. It should be emphasised that the telnet server can pass on the data it has received from the client to many other types of process including a remote login server. It is described in RFC854 and was first published in 1983.

The Network Virtual Terminal

Communication is established using the TCP/IP protocols and communication is based on a set of facilities known as a Network Virtual Terminal (NVT). At the user or client end the telnet client program is responsible for mapping incoming NVT codes to the actual codes needed to operate the user's display device and is also responsible for mapping user generated keyboard sequences into NVT sequences.
The NVT uses 7 bit codes for characters, the display device, referred to as a printer in the RFC, is only required to display the "standard" printing ASCII characters represented by 7 bit codes and to recognise and process certain control codes. The 7 bit characters are transmitted as 8 bit bytes with most significant bit set to zero. An end-of-line is transmitted as the character sequence CR (carriage return) followed by LF (line feed). If it is desired to transmit an actual carriage return this is transmitted as a carriage return followed by a NUL (all bits zero) character.
NVT ASCII is used by many other Internet protocols.
The following control codes are required to be understood by the Network Virtual Terminal.

Name code Decimal Value Function
NULL NUL 0 No operation
Line Feed LF 10 Moves the printer to the next print line, keeping the same horizontal position.
Carriage Return CR 13 Moves the printer to the left margin of the current line.
The following further control codes are optional but should have the indicated defined effect on the display.

Name code Decimal Value Function
BELL BEL 7 Produces an audible or visible signal (which does NOT move the print head.
Back Space BS 8 Moves the print head one character position towards the left margin. [On a printing devices this mechanism was commonly used to form composite characters by printing two basic characters on top of each other.]
Horizontal Tab HT 9 Moves the printer to the next horizontal tab stop. It remains unspecified how either party determines or establishes where such tab stops are located.
Vertical Tab VT 11 Moves the printer to the next vertical tab stop. It remains unspecified how either party determines or establishes where such tab stops are located.
Form Feed FF 12 Moves the printer to the top of the next page, keeping the same horizontal position. [On visual displays this commonly clears the screen and moves the cursor to the top left corner.]
The NVT keyboard is specified as being capable of generating all 128 ASCII codes by using keys, key combinations or key sequences.


The telnet protocol also specifies various commands that control the method and various details of the interaction between the client and server. These commands are incorporated within the data stream. The commands are distinguished by the use of various characters with the most significant bit set. Commands are always introduced by a character with the decimal code 255 known as an Interpret as command (IAC) character. The complete set of special characters is

Name Decimal Code Meaning
SE 240 End of subnegotiation parameters.
NOP 241 No operation
DM 242 Data mark. Indicates the position of a Synch event within the data stream. This should always be accompanied by a TCP urgent notification.
BRK 243 Break. Indicates that the "break" or "attention" key was hit.
IP 244 Suspend, interrupt or abort the process to which the NVT is connected.
AO 245 Abort output. Allows the current process to run to completion but do not send its output to the user.
AYT 246 Are you there. Send back to the NVT some visible evidence that the AYT was received.
EC 247 Erase character. The receiver should delete the last preceding undeleted character from the data stream.
EL 248 Erase line. Delete characters from the data stream back to but not including the previous CRLF.
GA 249 Go ahead. Used, under certain circumstances, to tell the other end that it can transmit.
SB 250 Subnegotiation of the indicated option follows.
WILL 251 Indicates the desire to begin performing, or confirmation that you are now performing, the indicated option.
WONT 252 Indicates the refusal to perform, or continue performing, the indicated option.
DO 253 Indicates the request that the other party perform, or confirmation that you are expecting the other party to perform, the indicated option.
DONT 254 Indicates the demand that the other party stop performing, or confirmation that you are no longer expecting the other party to perform, the indicated option.
IAC 255 Interpret as command
There are a variety of options that can be negotiated between a telnet client and server using commands at any stage during the connection.

Common Telnet options:

Decimal code Option Name RFC
0 Transmit Binary 856
1 Echo 857
3 Suppress Go Ahead 858
5 Status 859
6 Timing Mark 860
24 Terminal Type 1091
31 Window Size 1073
32 Terminal Speed 1079
33 Remote Flow Control 1372
34 Linemode 1184
36 Environment Variables 1408
All Telnet options:
Decimal Code Option Name RFC
0 Transmit Binary 856
1 Echo 857
2 Reconnection
3 Suppress Go Ahead 858
4 Approx Message Size Negotiation.
5 Status 859
6 Timing Mark 860
7 Remote Controlled Trans and Echo 563, 726
8 Output Line Width
9 Output Page Size
10 Negotiate About Output Carriage-Return Disposition 652
11 Negotiate About Output Horizontal Tabstops 653
12 NAOHTD, Negotiate About Output Horizontal Tab Disposition 654
13 Negotiate About Output Formfeed Disposition 655
14 Negotiate About Vertical Tabstops 656
15 Negotiate About Output Vertcial Tab Disposition 657
16 Negotiate About Output Linefeed Disposition 658
17 Extended ASCII. 698
18 Logout. 727
19 Byte Macro 735
20 Data Entry Terminal 732,1043
21 SUPDUP 734, 736
22 SUPDUP Output 749
23 Send Location 779
24 Terminal Type 1091
25 End of Record 885
26 TACACS User Identification 927
27 Output Marking 933
28 TTYLOC, Terminal Location Number. 946
29 Telnet 3270 Regime 1041
30 X.3 PAD. 1053
31 NAWS, Negotiate About Window Size. 1073
32 Terminal Speed 1079
33 Remote Flow Control 1372
34 Linemode 1184
35 X Display Location. 1096
36 Environment 1408
37 Authentication 1416, 2941, 2942, 2943,2951
38 Encryption Option 2946
39 New Environment 1572
40 TN3270E 2355
42 CHARSET 2066
43 RSP, Telnet Remote Serial Port
44 Com Port Control 2217
45 Telnet Suppress Local Echo
46 Telnet Start TLS
47 KERMIT 2840
255 Extended-Options-List RFC 861
Options are agreed by a process of negotiation which results in the client and server having a common view of various extra capabilities that affect the interchange and the operation of applications.
Either end of a telnet dialogue can enable or disable an option either locally or remotely. The initiator sends a 3 byte command of the form

 IAC,<type of operation>,<option>
The response is of the same form.
Operation is one of

Description Decimal Code Action
WILL 251 Sender wants to do something.
WONT 252 Sender doesn't want to do something.
DO 253 Sender wants the other end to do something.
DONT 254 Sender wants the other not to do something.
Associated with each of the these there are various possible responses

Sender Sent Receiver Responds Implication
WILL DO The sender would like to use a certain facility if the receiver can handle it. Option is now in effect
WILL DONT Receiver says it cannot support the option. Option is not in effect.
DO WILL The sender says it can handle traffic from the sender if the sender wishes to use a certain option. Option is now in effect.
DO WONT Receiver says it cannot support the option. Option is not in effect.
WONT DONT Option disabled. DONT is only valid response.
DONT WONT Option disabled. WONT is only valid response.
For example if the sender wants the other end to suppress go-ahead it would send the byte sequence


The final byte of the three byte sequence identifies the required action. For some of the negotiable options values need to be communicated once support of the option has been agreed. This is done using sub-option negotiation. Values are communicated via an exchange of value query commands and responses in the following form.

 IAC,SB,<option code number>,1,IAC,SE

IAC,SB,<option code>,0,<value>,IAC,SE
For example if the client wishes to identify the terminal type to the server the following exchange might take place

Client   255(IAC),251(WILL),24
Server   255(IAC),253(DO),24
Server   255(IAC),250(SB),24,1,255(IAC),240(SE)
Client   255(IAC),250(SB),24,0,'V','T','2','2','0',255(IAC),240(SE)
The first exchange establishes that terminal type (option number 24) will be handled, the server then enquires of the client what value it wishes to associate with the terminal type. The sequence SB,24,1 implies sub-option negotiation for option type 24, value required (1). The IAC,SE sequence indicates the end of this request. The repsonse IAC,SB,24,0,'V'... implies sub-option negotiation for option type 24, value supplied (0), the IAC,SE sequence indicates the end of the response (and the supplied value). The encoding of the value is specific to the option but a sequence of characters, as shown above, is common.