Meeting-20100612
WPLUG will have a General User Meeting and presentation on Saturday, June 12th, 2010, starting at 11am at the Wilkins School Community Center.
Schedule for the Day
10:30am - Doors open, set up
11:00am - Business Meeting starts
11:30am - Featured Presentation
12:30pm - Meeting ends, everyone out. We are likely to go to D's 6pack or Square Cafe for lunch.
Speaker/Presentation
Vance Kochenderfer will be talking a bit about the UNIX text processing utilities such as grep, sed, awk, cat, wc, and the like.
However, you don't get to just sit on your butt and listen; this is an audience-participation event. What we're going to do is take a couple simple tasks, and then explore how you could accomplish them using various UNIX utilities.
The goal is not solely to find the standard, quickest, or simplest solution, but to try out as many different whacked-out options as we can. So don't stop thinking once you've got an answer, even if it's a good one - see what else you can come up with!
We'll talk over all the suggestions and how they work (or don't work), so hopefully we'll all learn something new.
Start thinking about these, and bring your ideas to the meeting:
EXERCISE ONE
You have a large text file. Some lines contain text; others are blank. Your goal is to figure out how many non-blank lines are in the file.
I can think of at least six ways of doing this, how about you?
Below are the various examples I came up with, and running times using an input file with 1.5 million blank lines and 3 million non-blank lines. You can generate these statistics by preceding the command with 'time -p'.
# First example from <http://www.vectorsite.net/tsawk_3.html> awk 'NF != 0 { ++count } END { print count }' filename 3000000 real 3.67 user 3.55 sys 0.10
awk '/./ { ++count } END { print count }' filename
3000000
real 2.96
user 2.81
sys 0.12
grep -c . filename 3000000 real 0.94 user 0.84 sys 0.08
# sed is just a slower grep here. sed -n -e '/./p' filename | wc -l real 6.00 user 5.64 sys 0.14 3000000
# If you REALLY love sed, you can replace wc -l, too! sed -n -e '/./p' filename | sed -n -e '$=' real 7.43 user 5.70 sys 0.19 3000000
tr -s '\012' < filename | wc -l real 1.21 user 0.84 sys 0.13 3000000
# -b and -s are non-POSIX extensions to cat found on GNU and # BSD systems. cat -b -s filename | tail -n 2 | cut -f 1 real 1.20 user 0.63 sys 0.16 2999999 3000000
sh -c 'count=0 while read ln ; do [ -n "$ln" ] && count=$(($count+1)) done echo $count' < filename 3000000 real 240.14 user 214.12 sys 24.96
perl -e 'while (<>) { chomp; if ($_) { ++$count } } ;
print "$count\n"' < filename
3000000
real 7.22
user 7.00
sys 0.14
perl -e 'while (<>) { if (/./) { ++$count } } ;
print "$count\n"' < filename
3000000
real 8.93
user 8.78
sys 0.11
# This one displays a separate count of blank, non-blank, and
# total lines.
awk 'NF != 0 {++nonblank} NF == 0 {++blank}
END {print "Non-blank:",nonblank ; print "Blank:",blank ;
print "Total:",NR}' filename
Non-blank: 3000000
Blank: 1500000
Total: 4500000
real 5.42
user 5.28
sys 0.12
# Actually, we don't need a separate pattern and action to
# count blank lines; we can subtract from the total instead.
awk 'NF != 0 {++count}
END {print "Non-blank:",count ; print "Blank:",NR-count ;
print "Total:",NR}' filename
Non-blank: 3000000
Blank: 1500000
Total: 4500000
real 3.75
user 3.64
sys 0.09
# This does the same, but has to read the file three separate # times. On your system, might be faster or slower than the # one above; depends on whether CPU or I/O is the bottleneck. sh -c 'printf "Non-blank: " ; grep -c . filename ; printf "Blank: " ; grep -v -c . filename ; printf "Total: " ; wc -l filename | cut -d " " -f 1' Non-blank: 3000000 Blank: 1500000 Total: 4500000 real 2.25 user 1.91 sys 0.31
EXERCISE TWO
Determine whether a given value is numeric (decimal).
Example numeric values:
123 45.6789 -3.4567 -0 000123 .01234 54321. 00000. -0.987 -.987 -0123. 012 0.0 .0 -.000
Example non-numeric values:
hello 3f 3F AB 0xAB 0.0. -0- 3.0E8 3.0e-08 .-0123 1.23.4 5.678- --98 -. a space a tab
As a bonus, make your command also consider a value numeric if it starts with a + instead of a -.
I haven't thought about this one as much, and only have one solution so far. Maybe you can come up with something using bc or some other non-obvious method?
Answer: I wasn't able to find a way of doing this using bc or shell arithmetic as I thought I might. So the fallback was to use grep and build an appropriate regular expression (regex).
We ran out of time before I could explain the full regex, so here is the command in all its glory:
egrep -q '^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$'
or, if you want to strictly conform to POSIX,
grep -E '^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$' > /dev/null
We are using an extended regex, so we have to use egrep or the -E option to grep. Since we want to get a true/false value, we use the -q option or redirection to throw away the output. This way, we can just act based on grep's exit status.
Let's pick apart the regex to see what it does.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- ^
- The ^ at the beginning means that our regex will only match starting at the beginning of the line. The anticipated use for this command is something like 'echo "$value" | grep blah...', so we know there won't be any extraneous stuff at the beginning. If you are getting the value from a file or as input from the user, you may need to strip away whitespace from the beginning, or alter the regex to account for that.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- [-+]
- An expression inside brackets matches a single character, as long as that character is one of those listed inside the brackets (a range of characters can be specified, as can special pre-defined character classes, but in this case we're not using either of those). So this would match either a - sign or a + sign. Note that inside a bracket expression, + has no special meaning.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- [-+]?
- The question mark means "match zero or one of the preceding character." This makes it so that having a sign at the beginning of our number is optional, and also disallows multiple sign characters.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- ( ... | ... )
- The parentheses are there for grouping what's inside as a single unit. The pipe symbolizes alternation - that is, this part of the regex will match either the expression appearing before the pipe symbol, or the expression appearing after it. We need to use alternation because while it is optional to have numbers before the decimal point (e.g., .123) or after the decimal point (e.g., 123.), it is not valid for both sets of numbers to be missing (e.g., just a .).
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- [0-9]
- Another bracket expression, this matches any single numeral (that is, any character in the range 0 through 9).
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- [0-9]+
- The + sign outside a bracket expression does have special meaning. It means "match one or more of the preceding character." So this will match one or more numerals, but not an empty string.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- \.
- The dot has the special meaning "match any character." If we want to literally match a period, we have to escape it with a backslash to remove its special meaning. Note that [.] would do the same thing, as dot has no special meaning inside a bracket expression.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- \.?
- Again, question mark means to match zero or one of the preceding character. This makes the decimal point optional.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- [0-9]+\.?
- This is the entire first expression of our alternation. It will match an integer of any length, optionally followed by a decimal point. So 0, 123, 123., 00000., 000123, and 123123123123123123123 would all match, but just a decimal point would not.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- [0-9]*
- We've seen [0-9] before, but the * is new. It is similar to +, but means "match zero or more of the preceding character." We use * instead of + because it's valid to have nothing in front of the decimal point (e.g., .123).
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- \.
- This matches a decimal point again, but note there is no question mark. In this expression, one single decimal point is mandatory.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- [0-9]+
- Again, this matches one or more numerals. Having numbers after the decimal point is not optional here.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- [0-9]*\.[0-9]+
- This is the full second expression of our alternation. It will match any floating point value, such as 0.123, 1.234, .123, 0098.6, 123.456, or 3.1415926535897932384626433832795028841971693993751.
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- ([0-9]+\.?|[0-9]*\.[0-9]+)
- This is the full non-sign part of our regex. As discussed above regarding alternation, it means "either an integer, optionally followed by a decimal point, or an optional set of numerals followed by a (mandatory) decimal point and one or more trailing numerals."
^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$
- $
- This is the counterpart to the ^ character, forcing a match at the end of the line. Putting the expression inside ^$ forbids any extraneous characters before or after our match.
As we talked about, we could use this regex in awk like so:
awk '/^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$/ {action-list}'
where the action list would be executed for each numeric value in input. Note that sed does NOT support the extended regexes understood by egrep or 'grep -E'. It only handles the basic regexes of standard grep, so we cannot use this regex with sed.
If you are having difficulty understanding any of this, try playing around with different input values and/or altering the regex to see what the different parts do. Ask questions on the main wplug mailing list if you get really stuck. Have fun!
Example output:
cat numbers 123 45.6789 -3.4567 -0 000123 .01234 54321. 00000. -0.987 -.987 -0123. 012 0.0 .0 -.000 cat nonnumbers hello 3f 3F AB 0xAB 0.0. -0- 3.0E8 3.0e-08 .-0123 1.23.4 5.678- --98 .- cat bonusnumbers 123 45.6789 +3.4567 +0 000123 .01234 54321. 00000. +0.987 +.987 +0123. 012 0.0 .0 +.000 cat numbers | while read value ; do echo "$value" | \ grep -E '^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$' > /dev/null \ && echo "Number" || echo "Not a number"; done \ | paste numbers - 123 Number 45.6789 Number -3.4567 Number -0 Number 000123 Number .01234 Number 54321. Number 00000. Number -0.987 Number -.987 Number -0123. Number 012 Number 0.0 Number .0 Number -.000 Number cat nonnumbers | while read value ; do echo "$value" | \ grep -E '^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$' > /dev/null \ && echo "Number" || echo "Not a number"; done \ | paste nonnumbers - hello Not a number 3f Not a number 3F Not a number AB Not a number 0xAB Not a number 0.0. Not a number -0- Not a number 3.0E8 Not a number 3.0e-08 Not a number .-0123 Not a number 1.23.4 Not a number 5.678- Not a number --98 Not a number .- Not a number Not a number Not a number cat bonusnumbers | while read value ; do echo "$value" | \ grep -E '^[-+]?([0-9]+\.?|[0-9]*\.[0-9]+)$' > /dev/null \ && echo "Number" || echo "Not a number"; done \ | paste bonusnumbers - 123 Number 45.6789 Number +3.4567 Number +0 Number 000123 Number .01234 Number 54321. Number 00000. Number +0.987 Number +.987 Number +0123. Number 012 Number 0.0 Number .0 Number +.000 Number
Meeting Minutes
DRAFT
The regular monthly meeting of the Western Pennsylvania Linux Users Group was held on Saturday, June 12, 2009, at 11:06 AM, at the Wilkins School Community Center, the regular presiding officer being in the chair. In the absence of the regular secretary, Vance Kochenderfer was elected to serve as secretary pro tem. The minutes of the October 31, 2009 meeting were approved as read.
The Treasurer reported that there is $760.80 in the checking account, $66 cash on hand in the refreshment fund, and $40 received in dues yet to be deposited.
The meeting adjourned at 11:09 AM.
Vance Kochenderfer
Secretary pro tem
DRAFT
Meeting Staff
If you would like to volunteer to assist with this meeting, please add your name to one or more of the categories below.
- Host: Your name here
- Co-Host: Your name here
- Donuts/Bagels: David Kraus
- Setup: David Kraus, Your name here
- Clean Up: David Kraus, Your name here
Carpooling
- Your name/location here