Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts

Efficient data transfer through zero copy



Efficient data transfer through zero copy
Many Web applications serve a significant amount of static content, which amounts to reading data off of a disk and writing the exact same data back to the response socket. This activity might appear to require relatively little CPU activity, but it's somewhat inefficient: the kernel reads the data off of disk and pushes it across the kernel-user boundary to the application, and then the application pushes it back across the kernel-user boundary to be written out to the socket. In effect, the application serves as an inefficient intermediary that gets the data from the disk file to the socket.
Each time data traverses the user-kernel boundary, it must be copied, which consumes CPU cycles and memory bandwidth.

Applications that use zero copy request that the kernel copy the data directly from the disk file to the socket, without going through the application. Zero copy greatly improves application performance and reduces the number of context switches between kernel and user mode.
The Java class libraries support zero copy on Linux and UNIX systems through the transferTo() method in java.nio.channels.FileChannel. You can use the transferTo() method to transfer bytes directly from the channel on which it is invoked to another writable byte channel, without requiring data to flow through the application.

If you re-examine the traditional scenario, you'll notice that the second and third data copies are not actually required. The application does nothing other than cache the data and transfer it back to the socket buffer. Instead, the data could be transferred directly from the read buffer to the socket buffer.

The transferTo() method transfers data from the file channel to the given writable byte channel. Internally, it depends on the underlying operating system's support for zero copy; in UNIX and various flavors of Linux, this call is routed to the sendfile() system call, which transfers data from one file descriptor to another:

Listing 1. Copying bytes from a file to a socket
File.read(fileDesc, buf, len);
Socket.send(socket, buf, len);
Although Listing 1 is conceptually simple, internally, the copy operation requires four context switches between user mode and kernel mode, and the data is copied four times before the operation is complete.
Figure 1. Traditional data copying approach
Traditional data copying approach
Figure 2 shows the context switching:
Figure 2. Traditional context switches
Traditional context switches


Figure 3. Data copy with transferTo()
Data copy with transferTo()
Figure 4 shows the context switches when the transferTo() method is used:
Figure 4. Context switching with transferTo()
Context switching when using transferTo()
The transferTo() method causes the file contents to be copied into a read buffer by the DMA engine. Then the data is copied by the kernel into the kernel buffer associated with the output socket.
The third copy happens as the DMA engine passes the data from the kernel socket buffers to the protocol engine.
This is an improvement: we've reduced the number of context switches from four to two and reduced the number of data copies from four to three (only one of which involves the CPU). But this does not yet get us to our goal of zero copy. We can further reduce the data duplication done by the kernel if the underlying network interface card supports gather operations. In Linux kernels 2.4 and later, the socket buffer descriptor was modified to accommodate this requirement. This approach not only reduces multiple context switches but also eliminates the duplicated data copies that require CPU involvement. The user-side usage still remains the same, but the intrinsics have changed:
The transferTo() method causes the file contents to be copied into a kernel buffer by the DMA engine.
No data is copied into the socket buffer. Instead, only descriptors with information about the location and length of the data are appended to the socket buffer. The DMA engine passes data directly from the kernel buffer to the protocol engine, thus eliminating the remaining final CPU copy.
Figure 5 shows the data copies using transferTo() with the gather operation:
Figure 5. Data copies when transferTo() and gather operations are used
Data copies when transferTo() and gather operations are used

Read full article from Efficient data transfer through zero copy

Algorithms and Me: Anatomy of a Process



A process is a program in execution, with it associated are process context and executable instruction.
A process has its own data, code, stack, register and memory space. Every process has its own virtual memory address range, I/O resources, opened files etc.

Creation of a process
Most widely used method to create a process is to use 'fork' and 'exec' system calls. As mentioned earlier, every process has parent, parent uses fork system call to create exactly same copy of itself. Once new process is scheduled, it can use exec system call to execute any program it wants to.

fork is a call where one process goes in and two come out. They both start there execution from the statement just after fork call (Remember new process is exact copy, hence its PC will be same).

How to distinguish between parent and child process? fork comes to rescue there. Call to 'fork' return child process's PID to parent process while zero to child process. By having check on return value of 'fork' system call we can figure out which process is parent and which is child.

Now, fork can be a very expensive call as OS has to duplicate whole lot of information, especially the virtually memory and pages currently used by the parent process. There is one concept which is called 'Copy on Write', so fork system call will not copy any of the pages till the time one of the process tries to modify the page. This arrangement makes fork system call fast.


Other system call is exec(). It is used to start a new program, it will replace contents of process with of program binary. There are many versions of the same system call used for varying purposes.
  1. The calls with v in the name take an array parameter to specify the argv[] array of the new program.
  2. The calls with l in the name take the arguments of the new program as a variable-length argument list to the function itself.
  3. The calls with e in the name take an extra argument to provide the environment of the new program; otherwise, the program inherits the current process's environment.
  4. The calls with p in the name search the PATH environment variable to find the program if it doesn't have a directory in it (i.e. it doesn't contain a / character). Otherwise, the program name is always treated as a path to the executable
When a process creates a child process, it may or may not wait for return status of the child process.
To wait for the return status, parent process uses wait() system call. It blocks the parent process till the time one of its child returns status. Usually return status of child process is used to check if the child process terminated normally or abnormally. Child process can inform their exit status using SIGCHILD signal.
There are variants of wait() like wait3() and wait4() which are non blocking call on parent process.

Read full article from Algorithms and Me: Anatomy of a Process

What are the Zombie and the Orphan Processes and how to kill them? | LinuxG.net



the zombie (or defunctprocesses are dead processes that still apear in the process table, usually because of bugs and coding errors. A zombie process remains in the operating system and does nothing until the parent process determines that the exit status is no longer needed.

When does a process turn into a zombie?
Normally, when a process finishes execution, it reports the execution status to its parent process. Until the parent process decides that the child processes exit status is not needed anymore, the child process turns into a defunct or zombie process. It does not use resources and it cannot be schuduled for execution. Sometimes the parent process keeps the child in the zombie stateto ensure that the future children processes will not receive the same PID.

You can find the zombie processes with ps aux | grep Z. The processes with Z in the STATE field are zombie processes:
$ ps aux | grep Z

http://www.geekride.com/zombie-process-defunct-linux
Killing a Zombie Process:
Well, before taking any decision of killing the Zombie process, you should wait, as it is possible that the parent process is intentionally leaving the process in a zombie state to ensure that future children that it may create will not receive the same pid. Or perhaps the parent is occupied, and will reap the child process momentarily.

If that didn’t happen then you can send a SIGCHLD signal to the parent process of zombie which will instruct parents to reap their zombie children.

# kill -s SIGCHLD <PPID>
or kill  -17 <PPID>
Even if this don’t work, then the last option you will have is to kill the parent process. You can easily find out the parent’s process ID with this command:

# ps aux -eo ppid | grep <Zombie Process ID>
# kill -9 <PPID>
So when a Zombie process loses it’s parent process, it becomes orphan and adopted by “init”. Init periodically executes the wait system call to reap any zombies with init as parent.

QWhy I can’t kill a Zombie process with “kill” command ?
A. Zombie process is already dead, so killing them with “kill -9″ won’t help at all.
QIs it bad to have Zombie processes on your system ?
A. Well, as Zombie processes are not taking any resources of your system, leaving a small entry in process table, it’s not at all harmful to have Zombie processes in your system, but it may hurt you sometime under heavy load. So, it’s always better not to have them.
QIs Zombie process different from an Orphan process ?
A. Yes, Zombie is something which is already dead, but Orphan processes are those whose parents are dead.

Also refer to http://www.geekride.com/zombie-process-defunct-linux
Read full article from What are the Zombie and the Orphan Processes and how to kill them? | LinuxG.net

Orphan Process | Geek Ride




In Linux/Unix like operating systems, as soon as parents of any process are dead, re-parenting occurs, automatically. Re-parenting means processes whose parents are dead, means Orphaned processes, are immediately adopted by special process “init”.

A process can be orphaned either intentionally or unintentionally. Sometime a parent process exits/terminates or crashes leaving the child process still running, and then they become orphans.
Also, a process can be intentionally orphaned just to keep it running.
At the same time, when a client connects to a remote server and initiated a process, and due to some reason the client crashes unexpectedly, the process on the server becomes Orphan.

Finding a Orphan Process
ps -elf | head -1; ps -elf | awk '{if ($5 == 1 && $3 != "root") {print $0}}' | head
Is Orphan process different from an Zombie process ?
A. Yes, Orphan process are totally different from Zombie processes. Zombie processes are the ones which are not alive but still have entry in parent table.
Are Orphan processes harmful for system ?
A. Yes. Orphan processes take resources while they are in the system, and can potentially leave a server starved for resources. Having too many Orphan processes will overload the init process and can hang-up a Linux system. We can say that a normal user who has access to your Linux server is capable to easily kill your Linux server in a  minute.
Read full article from Orphan Process | Geek Ride

bash - Show only odd lines with cat - Super User



Show only odd lines
sed -n 1~2p filename
Using awk: awk 'NR % 2 == 0' filename
Read full article from bash - Show only odd lines with cat - Super User

Addresses - sed, a stream editor



Selecting lines with sed
number
first~step
This GNU extension matches every stepth line starting with line first. In particular, lines will be selected when there exists a non-negative n such that the current line-number equals first+ (n * step). Thus, to select the odd-numbered lines, one would use 1~2; to pick every third line starting with the second, ‘2~3’ would be used; to pick every fifth line starting with the tenth, use ‘10~5
$ - last line
/regexp/
0,/regexp/
addr1,+N
Matches addr1 and the N lines following addr1
addr1,~N
Matches addr1 and the lines following addr1 until the next line whose input line number is a multiple of N.
Appending the ! character to the end of an address specification negates the sense of the match. That is, if the ! character follows an address range, then only lines which do not match the address range will be selected. This also works for singleton addresses, and, perhaps perversely, for the null address.
Read full article from Addresses - sed, a stream editor

bash - How to keep only every nth line of a file - Super User



awk 'NR == 1 || NR % 3 == 0' yourfile
sed -n '1p;0~3p' input.txt
Read full article from bash - How to keep only every nth line of a file - Super User

How to Add Calculations to a Bash Script



let "m = 4 * 1024"; echo $m
let "m = a % 100"
let "m += 15"

let "m++"
let "m--"
let "k = (m < 9) ? 0 : 1"

Floating Point Arithmetic in Bash

The let operator only works for integer arithmetic. For floating point arithmetic you can use for example the GNU bc calculator:
echo "32.0 + 1.4" | bc 

Backticks (back single quotes) can be used to evaluate an arithmetic expression as in this example:
echo `expr $m + 18`

m=`expr $m + 18`
Another way to evaluate arithmetic expressions is to use double parenthesis:
(( m *= 4 ))
Read full article from How to Add Calculations to a Bash Script

How To Use the AWK language to Manipulate Text in Linux | DigitalOcean



The basic format:
awk 'BEGIN { action; }
/search/ { action; }
END { action; }' input_file

awk '/search_pattern/ { action_to_take_on_matches; another_action; }' file_to_parse
awk '/^UUID/ {print $1;}' /etc/fstab

Awk Internal Variables and Expanded Format
FILENAME: References the current input file.
FNR: References the number of the current record relative to the current input file. For instance, if you have two input files, this would tell you the record number of each file instead of as a total.
FS: The current field separator used to denote each field in a record. By default, this is set to whitespace.
NF: The number of fields in the current record.
NR: The number of the current record.
OFS: The field separator for the outputted data. By default, this is set to whitespace.
ORS: The record separator for the outputted data. By default, this is a newline character.
RS: The record separator used to distinguish separate records in the input file. By default, this is a newline character.

we can change some of the internal variables in the BEGIN section.
sudo awk 'BEGIN { FS=":"; }
{ print $1; }' /etc/passwd

awk '$2 ~ /^sa/' favorite_food.txt
awk '$2 !~ /^sa/' favorite_food.txt
awk '$2 !~ /^sa/ && $1 < 5' favorite_food.txt
Read full article from How To Use the AWK language to Manipulate Text in Linux | DigitalOcean

The Basics of Using the Sed Stream Editor to Manipulate Text in Linux | DigitalOcean



sed '' BSD
sed -n 'p' BSD

sed -n '1p' BSD
sed -n '1,5p' BSD
sed -n '1,+4p' BSD

If we want to print every other line, we can specify the interval after the "~" character. The following line will print every other line starting with line 1:

sed -n '1~2p' BSD

Deleting Text
delete every other line starting with the first:
sed '1~2d' BSD
sed -i '1~2d' everyother.txt
To create a backup file prior to editing, add the backup extension directly after the "-i" option:
sed -i.bak '1~2d' everyother.txt

Substituting Text
's/old_word/new_word/'
sed 's/on/forward/' annoying.txt
We will provide the "g" flag to the substitute command by placing it after the substitution set.
sed 's/on/forward/g' annoying.txt

If we only wanted to change the second instance of "on" that sed finds on each line, then we could use the number "2" instead of the "g".
sed 's/on/forward/2' annoying.txt

see which lines were substituted
sed -n 's/on/forward/2p' annoying.text

Ignore case:
sed 's/SINGING/saying/i' annoying.txt

Referencing Matched Text
put parentheses around the matched text:
sed 's/^.*at/(&)/' annoying.txt
sed 's/\([a-zA-Z0-9][a-zA-Z0-9]*\) \([a-zA-Z0-9][a-zA-Z0-9]*\)/\2 \1/' annoying.txt
sed 's/\([^ ][^ ]*\) \([^ ][^ ]*\)/\2 \1/' annoying.txt

How to keep only every nth line of a file

bash - How to keep only every nth line of a file - Super User
awk 'NR == 1 || NR % 3 == 0' yourfile
sed -n '1p;0~3p' input.txt
Read full article from The Basics of Using the Sed Stream Editor to Manipulate Text in Linux | DigitalOcean

Top 25 Unix interview questions with answers (Part I)



How to print/display the last line of a file?
tail -1 file.txt
sed -n '$ p' test

How to print/display the first line of a file?
head -1 file.txt
sed '2,$ d' file.txt
the 'd' parameter basically tells [sed] to delete all the records from display output from line no. 2 to last line of the file (last line is represented by $ symbol).

How to display n-th line of a file?
sed –n '<n> p' file.txt
head -<n> file.txt | tail -1

How to remove the first line?
sed '1 d' file.txt > new_file.txt
mv new_file.txt file.txt

Or, you can use an inbuilt [sed] switch '–i' which changes the file in-place.
sed –i '1 d' file.txt

How to remove the last line?
sed –i '$ d' file.txt

sed –i '5,7 d' file.txt

How to remove the last n-th line from a file?
tt=`wc -l a.txt | cut -f1 -d' '`;sed -i "`expr $tt - 4`,$tt d" a.txt

How to check the length of any line in a file?
sed -n '35 p' a.txt | wc -c

How to get the nth word of a line?
cut –f<n> -d' '
'-d' switch tells [cut] about what is the delimiter (or separator) in the file, which is space ' ' in this case. If the separator was comma, we could have written -d',' then.

echo "A quick brown fox jumped over the lazy cat" | cut -f4 -d' '

How to reverse a string?
echo "unix" | rev

How to get the last word from a line file?
echo "C for Cat" | rev | cut -f1 -d' ' | rev

wc -c file.txt | cut -d' ' -f1
wc -c a.txt | awk '{print $1}'

How to replace the n-th line in a file with a new line?
Step 1: remove the n-th line
$>sed -i'' '10 d' file.txt       # d stands for delete
Step 2: insert a new line at n-th line position
$>sed -i'' '10 i This is the new line' file.txt     # i stands for insert

How to test if a zip file is corrupted?
unzip –t file.zip

How to unzip a file?
unzip –j file.zip

How to check if a file is zipped ?
file a.txt
file -i a.txt

How to list down file/folder lists alphabetically?
[ls –lt] command lists down file/folder list sorted by modified time. If you want to list then alphabetically, then you should simply specify: [ls –l]

How to check if the last command was successful?
ls –l file.txt; echo $?

How to check all the running processes?
ps -e -o stime,user,pid,args,%mem,%cpu
By using “-o” switch, you can specify the columns that you want [ps] to print out.

ps –ef
If you wish to see the % of memory usage and CPU usage, then consider the below switches
ps aux

How to tell if my process is running?
ps -e -o stime,user,pid,args,%mem,%cpu | grep "opera"

How to get the CPU and Memory details in Linux server?
/proc/meminfo
/proc/cpuinfo

Read full article from Top 25 Unix interview questions with answers (Part I)

Basic vi Commands




To Exit vi
*:x<Return>quit vi, writing out modified file to file named in original invocation
 :wq<Return>quit vi, writing out modified file to file named in original invocation
 :q<Return>quit (or exit) vi
*:q!<Return>quit vi even though latest changes have not been saved for this vi call

Moving the Cursor
*j or <Return>
  [or down-arrow]
move cursor down one line
*k [or up-arrow]move cursor up one line
*h or <Backspace>
  [or left-arrow]
move cursor left one character
*l or <Space>
  [or right-arrow]
move cursor right one character
*0 (zero)move cursor to start of current line (the one with the cursor)
*$move cursor to end of current line
 wmove cursor to beginning of next word
 bmove cursor back to beginning of preceding word
 :0<Return> or 1Gmove cursor to first line in file
 :n<Return> or nGmove cursor to line n
 :$<Return> or Gmove cursor to last line in file

Inserting or Adding Text
*iinsert text before cursor, until <Esc> hit
 Iinsert text at beginning of current line, until <Esc> hit
*aappend text after cursor, until <Esc> hit
 Aappend text to end of current line, until <Esc> hit
*oopen and put text in a new line below current line, until <Esc> hit
*Oopen and put text in a new line above current line, until <Esc> hit

Changing Text
*rreplace single character under cursor (no <Esc> needed)
 Rreplace characters, starting with current cursor position, until <Esc> hit
 cwchange the current word with new text,
starting with the character under cursor, until <Esc> hit
 cNwchange N words beginning with character under cursor, until <Esc> hit;
  e.g., c5w changes 5 words
 Cchange (replace) the characters in the current line, until <Esc> hit
 ccchange (replace) the entire current line, stopping when <Esc> is hit
 Ncc or cNcchange (replace) the next N lines, starting with the current line,
stopping when <Esc> is hit

Saving and Reading Files

 :r filename<Return>read file named filename and insert after current line
(the line with cursor)
 :w<Return>write current contents to file named in original vi call
 :w newfile<Return>write current contents to a new file named newfile
 :12,35w smallfile<Return>write the contents of the lines numbered 12 through 35 to a new file named smallfile
 :w! prevfile<Return>write current contents over a pre-existing file named prevfile
Read full article from Basic vi Commands

Copy and paste text with vi or vim | a Tech-Recipes Tutorial



The command ‘Y’ or ‘yy’ copies (yanks) one or more lines. To copy one line, two lines, 10 lines, and all lines to the end of the file, respectively:
Y 2Y
10Y
yG

To paste the text contained in the buffer above (uppercase P) or below the current cursor position (lowercase p), respectively:
P p

It is also possible to yank text within a line. The following commands yank text from the current cursor position to the end of the word and the end of the line, respectively:
yw y$

Lower case p pastes after the cursor position and upper case P pastes before.

Paste will also work with deleted text, either lines or parts of lines. Be careful not to execute any other commands prior to pasting as this will empty the buffer.
Read full article from Copy and paste text with vi or vim | a Tech-Recipes Tutorial

vi/vim delete commands and examples | vi vim delete lines to end | alvinalexander.com



x   - delete current character
10x

dw  - delete current word
dd  - delete current line
5dd - delete five lines

d$  - delete to end of line
d0  - delete to beginning of line

:1,.d
delete to beginning of file

:.,$d
delete to end of file

If all lines in the file are to be deleted, this vi command specifies the range of deletion:
:1,$d
Read full article from vi/vim delete commands and examples | vi vim delete lines to end | alvinalexander.com

13 Vi Editor Interview Questions And Answers



/text: it will search for the string. after pressing enter it takes u to that text location.
How to enter from command mode to insertion mode using Vi Editor?
Ans: a,i
What is the difference between ZZ and :wq commands in Vi Editor?
ZZ is the command mode comand in uix to save and quit file.
:wq is the execute command mode command to save and quit file.

How to append a file to current file using Vi Editor?
:r file2
If you are working in file1 and want to append file2, than place the cursor where you want to append the new file and use the following command<
How to go 10 number line in command mode directly ?
:10

Which command is used to replace many characters in Vi Editor?
change command can be used to change a word/line.
cw change word forward
cb change word backward
c$ change from cursor to end of line
cL change from current line to and of screen
cG change from current line to and of file

or if you want to replace all occurence of some specific character
:s/oldText/newText/g

what are the different modes in vi editor?
Command mode
This is the default when you enter vi. In command mode, most letters, or short sequences of letters, that you type will be interpreted as command. If you press Esc when you're in command mode, your terminal will beep at you. This is a very good way to tell when you're in command mode.

Insert mode
In insert mode, whatever you type is inserted in the file at the cursor position. Type a (lowercase letter a, for append) to enter insert mode from command mode; press Esc
to end insert mode, and return to command mode.

Last Line mode
Use line mode to enter line oriented commands. To enter line mode from command mode, type a colon ( : ). Your cursor moves to the bottom of the screen, by a colon prompt. Type a line mode command, then press Enter. Any sensible command from the Unix line editor ex will work, and a few are good to know about. These commands are indicated in this handout by a colon in front of the command. Each time you use a line mode command, you must type a colon to enter line mode, then type the command by the colon prompt at the bottom of the screen, then press Enter when you finish typing the command. (The search commands starting with / and ? work similarly).

Read full article from 13 Vi Editor Interview Questions And Answers

Understanding UNIX / Linux filesystem Inodes - nixCraft



The inode identifies the file and its attributes (as above) . Each inode is identified by a unique inode number within the file system. Inode is also know as index number.
An inode is a data structure on a traditional Unix-style file system such as UFS or ext3. An inode stores basic information about a regular file, directory, or other file system object.
You can use ls -i command to see inode number of file
ls -i /etc/passwd
You can also use stat command to find out inode number and its attribute:
stat /etc/passwd

Inode application
Many commands used by system administrators in UNIX / Linux operating systems often give inode numbers to designate a file.
cd /tmp
touch \"la*
ls -l
Now try to remove file "la*
You can't, to remove files having created with control characters or characters which are unable to be input on a keyboard or special character such as ?, * ^ etc. You have to use inode number to remove file.
Read full article from Understanding UNIX / Linux filesystem Inodes - nixCraft

10 xargs command example in Linux - Unix tutorial



with and without xargs
you can clearly see that multiline output is converted into single line:
find . -name "*bash*" | xargs
xargs and grep
find . -name "*.java" | xargs grep "Stock"
delete temporary file using find and xargs
find /tmp -name "*.tmp" | xargs rm
xargs -0 to handle space in file name
find /tmp -name "*.tmp" -print0 | xargs -0 rm
xargs and cut command in Unix
cut -d, -f1 smartphones.csv | sort | xargs
Counting number of lines in each file using xargs and find
ls -1 *.txt | xargs wc -l
Passing subset of arguments to xargs in Linux.
when used with xargs you can use flag "-n" to instruct xargs on how many argument it should pass to given command. this xargs command line option is extremely useful on certain situation like repeatedly doing diff etc
ls -1 *.txt | xargs -n 2 echo
avoid "Argument list too long"
xargs in unix or Linux was initially use to avoid "Argument list too long" errors and by using xargs you send sub-list to any command which is shorter than "ARG_MAX" and that's how xargs avoid "Argument list too long" error. You can see current value of "ARG_MAX" by using getconf ARG_MAX.

find –exec vs find + xargs
xargs with find command is much faster than using -exec on find. since -exec runs for each file while xargs operates on sub-list level. to give an example if you need to change permission of 10000 files xargs with find will be almost 10K time faster than find with -exec because xargs change permission of all file at once
Read full article from 10 xargs command example in Linux - Unix tutorial

xargs: How To Control and Use Command Line Arguments



utility. xargs reads items from the standard input or pipes, delimited by blanks or newlines, and executes the command one or more times with any initial-arguments followed by items read from standard input. Blank lines on the standard input are ignored.
echo 1 2 3 4 | xargs echo
Find all .bak files in or below the current directory and delete them.
find . -name "*.bak" -type f -print | xargs /bin/rm -f
{} as the argument list marker
{} is the default argument list marker. You need to use {} this with various command which take more than two arguments at a time. For example mv command need to know the file name. The following will find all .bak files in or below the current directory and move them to ~/.old.files directory:
find . -name "*.bak" -print0 | xargs -0 -I {} mv {} ~/old.files
You can rename {} to something else. In the following example {} is renamed as file. This is more readable as compare to previous example:
find . -name "*.bak" -print0 | xargs -0 -I file mv file ~/old.files
Avoiding errors and resource hungry problems with xargs and find combo
find /share/media/mp3/ -type f -name "*.mp3" -print0 | xargs -0 -r -I file cp -v -p file --target-directory=/bakup/iscsi/mp3

Reference: http://www.computerhope.com/unix/xargs.htm
--null, -0 Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (every character is taken literally). Disables the end-of-file string, which is treated like any other argument. Useful when input items might contain white space, quote marks, or backslashes. The find -print0 option produces input suitable for this mode.

find /tmp -name core -type f -print | xargs /bin/rm -f
Find files named core in or below the directory /tmp and delete them. (Note that this will work incorrectly if there are any filenames containing newlines or spaces.)
find /tmp -name core -type f -print0 | xargs -0 /bin/rm -f
find /tmp -depth -name core -type f -delete
Find files named core in or below the directory /tmp and delete them, but more efficiently than in the previous example (because we avoid the need to use fork and exec rm, and we don't need the extra xargs process).
cut -d: -f1 < /etc/passwd | sort | xargs echo
Uses cut to generate a compact listing of all the users on the system.

Read full article from xargs: How To Control and Use Command Line Arguments

10 Example of find command in Unix and Linux



How to find files which has been modified less than one day, minute or hour  in Unix:
find . -mtime 1  (find all the files modified exact 1 day)
find . -mtime -1 (find all the files modified less than 1 day)
find . -mtime +1 (find all the files modified more than 1 day)

How to find all the files and directories which holds the 777 permission
find . -perm 644

Case insensitive search using find
How to do case insensitive search using find command in Unix? Use option “-i" with name, by default find searches are case sensitive. This option of find is extremely helpful while looking for errors and exceptions in log file.
find . –iname "error" –print ( -i is for ignore )

How to find files based on size in Unix and Linux
find . -size +1000c -exec ls -l {} \;
find . -size +10000c -size -50000c -print
The minus sign means "less than," and the plus sign means "greater than."
This find example lists all files that are greater than 10,000 bytes, but less than 50,000 bytes.

How to find files some days older and above certain size
find . -mtime +10 -size +50000c -exec ls -l {} \;

How to use find command on file names with space
find . -name "*equity*" -print
find . -name "*equity*" -print | xargs ls -l
find . -name "*equity*" -print0 | xargs ls
Read full article from 10 Example of find command in Unix and Linux

Labels

Algorithm (219) Lucene (130) LeetCode (97) Database (36) Data Structure (33) text mining (28) Solr (27) java (27) Mathematical Algorithm (26) Difficult Algorithm (25) Logic Thinking (23) Puzzles (23) Bit Algorithms (22) Math (21) List (20) Dynamic Programming (19) Linux (19) Tree (18) Machine Learning (15) EPI (11) Queue (11) Smart Algorithm (11) Operating System (9) Java Basic (8) Recursive Algorithm (8) Stack (8) Eclipse (7) Scala (7) Tika (7) J2EE (6) Monitoring (6) Trie (6) Concurrency (5) Geometry Algorithm (5) Greedy Algorithm (5) Mahout (5) MySQL (5) xpost (5) C (4) Interview (4) Vi (4) regular expression (4) to-do (4) C++ (3) Chrome (3) Divide and Conquer (3) Graph Algorithm (3) Permutation (3) Powershell (3) Random (3) Segment Tree (3) UIMA (3) Union-Find (3) Video (3) Virtualization (3) Windows (3) XML (3) Advanced Data Structure (2) Android (2) Bash (2) Classic Algorithm (2) Debugging (2) Design Pattern (2) Google (2) Hadoop (2) Java Collections (2) Markov Chains (2) Probabilities (2) Shell (2) Site (2) Web Development (2) Workplace (2) angularjs (2) .Net (1) Amazon Interview (1) Android Studio (1) Array (1) Boilerpipe (1) Book Notes (1) ChromeOS (1) Chromebook (1) Codility (1) Desgin (1) Design (1) Divide and Conqure (1) GAE (1) Google Interview (1) Great Stuff (1) Hash (1) High Tech Companies (1) Improving (1) LifeTips (1) Maven (1) Network (1) Performance (1) Programming (1) Resources (1) Sampling (1) Sed (1) Smart Thinking (1) Sort (1) Spark (1) Stanford NLP (1) System Design (1) Trove (1) VIP (1) tools (1)

Popular Posts