sed

由 datahunter 在二, 20/12/2011 - 15:16 發表

最後更新: 2019-12-10

介紹

Sed (stream editor)

很詳細的 Documment: http://www.grymoire.com/Unix/Sed.html

常用 options
Command
刪除()
取代(substitution)
Use variable in replacement
"|" ":" "@"
print (簡寫 p)
Comments
Print line number with =
Transform with y (1對1的)
Grouping with { and }
Adding(a), Inserting(i), Changing lines
How sed Works
Writing a file with the 'w' command
Change configuration files setting value (ini file)

常用 options

sed [OPTION]... {script-only-if-no-other-script} [input-file]...

# version

sed --version

GNU sed version 4.2.1

-n, --quiet ( 在 print(p) 時只會 output 中 pattern 的 line )

-e 'script' # 可略

echo -e '1\n2\n3\n4\n5' > test.txt
sed '1d' test.txt               # 刪除第1行

-f script-file

echo 1d > cmd.sed
sed -f cmd.sed test.txt         # 刪除第1行

修改 File

當沒有 -i 或 -c 時, 只會在 stdout 輸出改過後的內容

# edit files in place (makes backup if extension supplied)

-i[SUFFIX], --in-place=[SUFFIX]

# use copy instead of rename when shuffling files in -i mode.

-c, --copy

# Use extended regular expressions in the script.

-r, --regexp-extended

Addresses

Syntax

addr1,addr2[!]

no addresses

in which case the command will be executed for all input lines;

one address

command will only be executed for input lines which match that address

two addresses # address-range

starting from the first address and continuing to the second address

Silent mode(-n)

cat test.txt

first
second
third

Prints only lines 2 through 3, as requested

sed -n '2,3p' test.txt

second
third

Prints each line (automatically), AND ALSO prints lines 2-3 a second time

By default, sed will print out the pattern space at the end of each cycle through the script.

These options(-n) disable this automatic printing,
and sed will only produce output when explicitly told to via the p command.

sed '2,3p' test.txt

first
second
second
third
third

"-n" 在 "s/" 情況

sed -n 's/t/T/' test.txt # replaces "t" with "T" on each line, but doesn't print the result due to -n

sed 's/t/T/' test.txt # replaces "t" with "T" on each line, and automatically prints the result

Command

s/regexp/replacement/ <--- 用得最多
p
d
w filename
y
=

Delimiter

/ _ : | @

一切為了簡化

將 path.txt 的內容由 /opt/name/bin 變成 /opt/tools/bin

sed 's/\/opt\/name\/bin/\/opt\/tools\/bin/g' path.txt

應用:

sed 's@/opt/name/bin@/opt/tools/bin@g' path.txt

實際使用

刪除

建立 test.txt

echo -e '1\n2\n\n4\n5' > test.txt

# 刪除第N行

# 刪除第2行

sed '2d' /etc/services

* 配合 "-i" 可原地 replace

# 刪除第2至第4行

sed '2,4d' test.txt

# 只保留第 2~4 行, 刪除其他

sed '2,4!d' test.txt

# Remove line "matched" by a regex

# 刪除有英文字"awk" 的所有行

sed '/awk/d' file.txt >  file-new.txt

# Remove all empty lines:

sed '/^$/d' filename.txt

取代(substitution)

把在 testfile 檔內的每行 is 取代成 are, 然後 output 到 stdout

sed 's/is/are/g' testfile

sed 's#is#are#g' testfile

Remark

- The s stands for substitute,

- the g stands for global. which means that all matching occurrences in the line would be replaced

有些版本的 sed 是不用加 /g 的也是 global replacement

直接 replace file

sed -i 's/is/are/g' testfile

Interesting usage insert / remove "#"

# insert "#" at the beginning of a line

sed -i '3 s/^/#/' test.txt

# remove "#" at the beginning of a line

sed -i '3 s/^#//' test.txt

Add string to each line in file

sed -e 's/$/string after each line/' -i filename

刪除 "\n"

情況

cat -A tmp.txt

CH1: OFF$
$
CH2: OFF$
$

sed 's#\n\n#\n#g' # replace 失敗 !!

原因

The "\n" will never match the newline at the end-of-line

because the newline is always stripped off before the line is placed into the pattern space.

解決方法

-z separate lines by NUL characters

sed -z 's#\n\n#\n#g'

刪除行首開始的 3 粒字

echo 123456 | sed 's%^.\{3\}%%g'

Use variable in replacement

RepoFile="CentOS-Base.repo"

BASEURL="https://repo.hlhk.net:10443/centos/centos6"

# 使用 (") 去引入 variable 及 (\$) 去 excape 其他 $

sed -i "\@^#baseurl=http.*/contrib/.*@a baseurl=$BASEURL/\$releasever/contrib/\$basearch/" $RepoFile

Regular Expression

Position

^ Matches the beginning of the line
$ Matches the end of the line

/^#/            Will match any line that begins with a '#'
/^$/            Will match all blank lines
/}$/            Will match any lines that ends with '}' (no spaces)
/} *$/          Will match any line ending with '}' followed by zero or more spaces

數量

* # match zero or more occurrences of the previous character
\+ # matches one or more
\? # matches zero or one.
\{i\} \{i,j\} \{i,\} # i 個; i~j 個; i 個或以上

字符

. Matches any single character <--- 真正的"." 是 "\."

/./             Will match any line that contains at least one character
/../            Will match any line that contains at least two characters

[] Matches all the characters inside the [ ]

/[abc]/         Will match any line that contains a lowercase 'a', 'b', or 'c'
/^[abc]/        Will match any line that begins with an 'a', 'b', or 'c'

Ampersand (&):

# 用 sed 扮 grep

cat file.txt

abcd
bcde
cdef

sed -n -e's/pattern/&/p' <file.txt

abcd
bcde

# Duplicate

echo "123 abc" | sed 's/[0-9]*/& &/'

123 123 abc

P.S.

"&" - corresponds to the pattern found

"(pattern)" - \1 through \9 to refer to the corresponding matching

Example

# -n option which tells sed not to print

# p flag which tells sed to print what is matched

ls -og /sys/block/sd*

lrwxrwxrwx 1 0 May 17 19:24 /sys/block/sda ->
 ../devices/pci0000:00/0000:00:11.0/ata1/host0/target0:0:0/0:0:0:0/block/sda

ls -l /sys/block/sd* | sed -n 's/.*$sd.$ -.*$ata.*$\/h.*/\2 => \1/p'

print (簡寫 p)

/p - print

sed -n -e '/regexp/p' /path/to/my/test/file | more

-n 只print 有修改的行

By default, sed prints every line. If it makes a substitution, the new text is printed instead of the old one.

找出檔案在 N 與 M之間的內容(N,M 可以是數字式 patten):

usage:

sed -n -e'N,Mp' file.txt

i.e.

sed -n -e'6,7p' test.txt

sed -n '/begin/,/end/p' test.txt

Comment

Sed comments are lines where the first non-white character is a "#."

On many systems, sed can have only one comment, and it must be the first line of the script.

Print line number with =

sed -n '/PATTERN/ =' file

Transform with y (1對1對換)

test.txt

aaaaaa11111
bbbbbb22222
cccccc33333
dddddd44444

sed 'y/bbbbbb/eeeeee/' test.txt

aaaaaa11111
eeeeee22222
cccccc33333
dddddd44444

b 與 e 必須一樣長度

Multiple commands for one address(Grouping with { and })

Sometimes, you may want to specify multiple commands that will apply to a single address.

This comes in especially handy when you are performing lots of 's///' to transform words or syntax in the source file.

To perform multiple commands per address, enter your sed commands in a file, and use the '{ }' characters to group commands, as follows:

i.e.

1,20{
    s/[Ll]inux/GNU\/Linux/g
    s/samba/Samba/g
    s/posix/POSIX/g
}

i.e.

1,/^END/{
    s/[Ll]inux/GNU\/Linux/g
    s/samba/Samba/g
    s/posix/POSIX/g
    p
}

This example will apply all the commands between '{ }' to the lines starting at 1 and up to a line beginning with the letters "END",

or the end of file if "END" is not found in the source file.

i.e.

#!/bin/sh
# This is a Bourne shell script that removes #-type comments
# between 'begin' and 'end' words.
sed -n '
    /begin/,/end/ {
         s/#.*//
         s/[ ^I]*$//
         /^$/ d
         p
    }
'

Append(a|r), Inserting(i), Changing(c) lines

Append a line with 'a'

Example 1: sed append to end of file

echo 'line 1' > test.txt

cat -A test.txt

line 1$

sed 'a line 2' test.txt # 相當於 echo 'line 2' >> test.txt

line 1
line 2

Example 2: Appends a line after the range or pattern.

This example will add a line after every line with "WORD"

#!/bin/sh
sed '
/WORD/ a\
Add this line after every line with WORD
'

* lines are continued by ending the previous line with a "\"

Append a file

r filename # Append text read from filename.

Insert a line with 'i'

You can insert a new line before the pattern with the "i" command:

#!/bin/sh
sed '
/WORD/ i\
Add this line before every line with WORD
'

在 match 的行的行頭 insert "#"

Usage:

sed /match/s/^/#/ filename

i.e.

sed /authn_dbd_module/s/^/#/ 00-base.conf

Change a line with 'c'

You can change the current line with a new line.

#!/bin/sh
sed '
/WORD/ c\
Replace the current line with the line
'

i.e.

cat myfile

aaaaaa11111
bbbbbb22222
cccccc33333
dddddd44444

sed '/bbbbbb/c eeeee' myfile

cat myfile

aaaaaa11111
eeeee
cccccc33333
dddddd44444

A "d" command followed by a "a" command won't work, as I discussed earlier.

The "d" command would terminate the current actions. You can combine all three actions using curly braces:

#!/bin/sh
sed '
/WORD/ {
i\
Add this line before
a\
Add this line after
c\
Change the line to this one
}'

How sed Works

Actually, if sed prints a line without the terminating newline, it will nevertheless print the missing newline as soon as more text is sent to the same output stream, which gives the “least expected surprise” even though it does not make commands like ‘sed -n p’ exactly identical to cat.

sed maintains two data buffers: the active pattern space, and the auxiliary hold space. Both are initially empty.

sed operates by performing the following cycle on each line of input: first, sed reads one line from the input stream, removes any trailing newline, and places it in the pattern space. Then commands are executed;

When the end of the script is reached, unless the -n option is in use, the contents of pattern space are printed out to the output stream, adding back the trailing newline if it was removed.

Then the next cycle starts for the next input line.

Pattern Space & Hold Space

Pattern Space:

The pattern space is the buffer where sed reads and processes each line of input.

It is a temporary buffer where sed reads and performs operations on the input text.

Hold Space:

The hold space is an additional buffer that can be used to store content temporarily.

It is not affected by the commands executed on the pattern space.

The hold space is useful when you need to perform operations that involve multiple lines or require storing data for later use.

Multiline techniques

D,G,H,N,P are similar to their lowercase counterparts (d,g, h,n,p),
except that these commands append or subtract data while respecting embedded newlines

To transfer data between the pattern space and the hold space,
you can use the h and g commands.
h copies the pattern space to the hold space, while g copies the hold space to the pattern space.

Appends line from the input file to the pattern space.

When sed reads a line of input, it stores it in the pattern space.

By default,sed processes one line at a time,
reading the input line, performing any specified operations on it,
and then outputting the modified line.

i.e.

file.txt content

=============
@line1 ABCDE@
@line2 FGHIJ@
@line3 KLMNO@
=============

# N appends the next line to the pattern space.
# By using N multiple times, we can append multiple lines to the pattern space.

sed -n '/ABCDE/{N;N;p;}' file.txt

appends line from the pattern space to the hold space, with a newline before it.

appends line from the hold space to the pattern space, with a newline before it.

deletes line from the pattern space until the first newline, and restarts the cycle.

prints line from the pattern space until the first newline.

Example

# h, H, G

seq 6 | sed '1h; 2,4H; 5G'

seq 6 | sed -n 'N;l;D'

The l command prints the content of the pattern space unambiguously ("\n")

The D command then removes the content of pattern space up to the first newline

(leaving ‘2’ at the end of the cycle).

進階 Search & Replace

Apply the replacement to all matches to the regexp, not just the first.

number

Only replace the numberth match of the regexp.

If the substitution was made, then print the new pattern space.

The m modifier to regular-expression matching is a GNU sed extension
which directs GNU sed to match the regular expression in multi-line mode.
There are special character sequences (\` and \') which always match the beginning or the end of the buffer.
In addition, the period character does not match a new-line character in multi-line mode.

# 解決 ^ 不是每行的情況

sed '/ABCDE/{N;N;s/^@//g;}' file.txt

 line1 ABCDE
@ line2 FGHIJ
@ line3 KLMNO

w filename

If the substitution was made, then write out the result to the named file.
As a GNU sed extension, two special values of filename are supported: /dev/stderr.

BRE & ERE

In GNU sed, the only difference between basic and extended regular expressions
is in the behavior of a few special characters: ‘?’, ‘+’, parentheses, braces (‘{}’), and ‘|’.

With basic (BRE) syntax, these characters do not have special meaning unless prefixed with a backslash (‘\’);
While with extended (ERE) syntax it is reversed: these characters are special unless they are prefixed with backslash (‘\’).

Example

BRE(Default)

echo 'a+b=c' | sed -n '/a+b/p'

ERE("-E")

echo 'a+b=c' | sed -E -n '/a\+b/p'

Writing a file with the 'w' command

sed -n 's/^[0-9]*[02468] /&/w even' < file

I used the "&" in the replacement part of the substitution command so that the line would not be changed.

A simpler example is to use the "w" command, which has the same syntax as the "w" flag in the substitute command

Change configuration files setting value (ini file)

cat test.ini

[Default]
setting1=false
setting2=false

方法1: 直觀的 search & replace

# -r regex

replaceValue=true
sed -i -r "s/^setting1=.*/setting1=$replaceValue/" test.ini

方法2: 行 script

# 在中 "^setting1=" 那行上將 "=" 及其後的字取代成 "=true"

replaceValue=true
sed -i -e "/^setting1=/ s/=.*/=$replaceValue/" test.ini

方法3: 不建議, 因為有機會有多個 Section

sed -i "/^setting2=/d" test.ini
echo "setting2=$replaceValue" >> test.ini

Scripts

set_config.sh

#!/bin/bash

set_config(){
        if [ -f $1 ]; then
                sed -i -r "s@^$2\s*=.*@$2=$3@" $1
        else
                echo "No file $1"
        fi
}

set_config php.ini short_open_tag On
set_config php.ini expose_php Off
set_config php.ini memory_limit 256M
set_config php.ini max_execution_time 60
set_config php.ini session.gc_maxlifetime 3600
set_config php.ini upload_max_filesize 80M
set_config php.ini post_max_size 100M

增加新的回應
瀏覽次數： 81762

夢想家

sed

介紹

目錄

常用 options

Addresses

Silent mode(-n)

Command

Delimiter

實際使用

刪除

取代(substitution)

Regular Expression

print (簡寫 p)

Comment

Print line number with =

Transform with y (1對1對換)

Multiple commands for one address(Grouping with { and })

Append(a|r), Inserting(i), Changing(c) lines

How sed Works

Pattern Space & Hold Space

Multiline techniques

進階 Search & Replace

BRE & ERE

Writing a file with the 'w' command

Change configuration files setting value (ini file)