Here’s some notable cli entries that I refer to occassionally:

Mac OS X:

sudo /usr/sbin/sysctl -w net.inet.ip.fw.enable=1
sudo /sbin/ipfw -q /etc/firewall.conf
sudo ifconfig en0 lladdr 00:1e:c2:0f:86:10
sudo route add -net 10.2.1.0/24 10.3.1.1

Linux:

So last weekend, I discovered that Spamhaus decided it would be a good idea to place all of the public IP addresses for Slicehost (my Linux VPS hoster) into their Spamhaus block list (SBL). This covered both my slice in Dallas and the one in St. Louis – meaning an impressive chunk of inbound mail to my domains was being trashed by the sending MTA and an even bigger chunk of my outbound mail was being outright rejected since the sending IP’s were on the SBL.  Slicehost worked hard to convince Spamhaus to recind the blocklist, so the Slicehost IP’s got moved over to the less-nasty-but-you’re-still-probably-a-spamming-dirtbag Policy Block list (PBL) assuming affected IP owners would request to be removed from that list.

Sample query to see if you’re on any Spamhaus block list:  http://www.spamhaus.org/query/bl?ip=10.11.12.13

It seems it’s time to relinquish the care and feeding of my own Postfix mail system and turn to a hosted solution.  This means I need to migrate about 5GB of IMAP store to another site (again).  Last time I did a wholesale migration, I used imapsync to make the transition painless. In the code example below, an SSL connection to the IMAPS server at imap-server.sourcedomain.com is made with username@sourcedomain.com and the password stored in the plaintext file secret1. An SSL connection is made to the target system (which happens to be the server on which the imapsync tool is running, but could just as easily be another IMAPS server somewhere on a network accessible to the host where imapsync is running). The –delete and –expunge1 arguments will clean the successfully moved messages from IMAP store #1 .. so be sure you have your messages on the target successfully! Imapsync can be run iteratively to ensure you have got all the messages from your source.


/usr/bin/imapsync \
--host1 imap-server.sourcedomain.com \
--ssl1 \
--authmech1 LOGIN \
--user1 username@sourcedomain.com --passfile1 secret1 \
--host2 127.0.0.1 --user2 username@targetdomain.com --passfile2 secret2 \
--ssl2 \
--delete --expunge1 \
--buffersize=128

And one can use the

--dry

option to just test the process but not actually move any of the messages.

So that’s it – I’m about half way though migrating my current IMAP stores over to a hosted mail solution, so that I don’t need to keep up with the increasing level of care and feeding that running your own mail service requires.  Before I get too many darts about that .. I first started running my own personal MTA in 1995, adding spam and av filtering over time, and adding substantial redundancy (servers, sites, storage) so I could rely on it and fix things that broke as I had time rather than right when they broke (which was always at a bad time).  My new hosted solution takes over from two VPS servers running Postfix, Spamassassin, ClamAV, Greylisting with the IMAP store replicated across data centers in different states (15 minute rsyncs).  So soon, the (hopefully) last Allen Pomeroy owned and operated MTA can be turned off, while I get to work on fun stuff, rather than figuring out why my email is bouncing.  :-)

Recently I found myself in the unhappy position of needing to sift through slightly more than a billion Checkpoint Firewall-1 log lines, looking for specific patterns of access. The problem was that many of the exported fwm log files had differing column positions and there had been many ruleset changes over the course of 11 months worth of log data. Many of the excellent FW1 log summarization tools (such as Peter Sundstrom’s fwlogsum) didn’t handle the hundreds of files and differing column positions.

The final scripted solution was processing over 11,000 lines/second .. and still took over 23 hours for the first run.

Log file exports via fwm logexport can have variable column positioning, except for record ID number “num”, which is *always* column number one.  I see three viable alternatives to the changing column position in the ASCII log files exported via fwm – so we can automate the log processing:

1. Parse the header line (line #1) of every log file and dynamically map (rearrange) the columns to a pre-determined standard in memory before further processing (painful, expensive)
2. Tell Checkpoint fwm to export in a fixed column ordering
   create
   logexport.ini
   and place in
   $FWDIR/conf directory
   eg. fwmgmtsrv:
   C:\WINDOWS\FW1\R65\FW1\conf
   logexport.ini:
   [Fields_Info]
included_fields = num,date,time,orig,origin_id,type,action,alert,i/f_name,
i/f_dir,product,rule,src,dst,proto,service,s_port,xlatesrc,xlatedst,
nat_rulenum,nat_addtnl_rulenum,xlatesport,xlatedport,user,
partner,community,session_id,ipv6_src,ipv6_dst,
srckeyid,dstkeyid,CookieI,CookieR,msgid,elapsed,
bytes,packets,start_time,snid,ua_snid,d_name,id_src,ua_operation,
sso_type_desc,app_name,auth_domain,uname4domain,wa_headers,
result_desc,r_dest,comment,url,redirect_url,enc_desc,e2e_enc_desc,
auth_result,attack,log_sys_message,
rule_uid,rule_name,service_id,resource,reason,cat_server,
dstname,SOAP Method,category,ICMP,message_info,
TCP flags,rpc_prog,Total logs,
Suppressed logs,DCE-RPC Interface UUID,Packet info,
message,ip_id,ip_len,ip_offset,fragments_dropped,during_sec
3. Use OPSEC LEA tools to extract event log records instead of export via fwm logexport

Once the ASCII log files are available for processing, my fw1logsearch.pl script can be used to find complex patterns of interest.  Any matching records found by fw1logsearch will be output with an initial FW1 header line so that fw1logsearch can be used iteratively, to build very complex search criteria.  fw1logsearch can also write out a discard file allowing completely negative logic searches resulting in 100% of the input data separated into a match file and a didn’t match file.  Some examples of how I’ve used it are shown here:

gunzip -c fwlogs/2009*gz | \
fw1logsearch.pl --allinclude \
-S '10\.1\.1[1359]\.|10\.2\.1[01]\.|192\.168\.2[245]\.' \
-d '10\.1\.1[1359]\.|10\.2\.1[01]\.|192\.168\.2[245]\.' \
-p '^1310$|^1411$|^1812$|^455' | \
fw1logsearch.pl -S '192\.168\.22\.14$|10\.2\.11\.12$' |\
fw1logsearch.pl --allexclude \
-S '^192\.168\.24\.12$' -P '^1310$' --rejectfile 192-168-24-12-port-1310.txt

Line by line:
1. Unzip the compressed ASCII log files, feed them to the first instance of fw1logsearch.pl
2. First fw1logsearch – all conditions must be true for any events to match
Source address must NOT be in any of the following regex ranges:
10.1.11.* 10.1.13.* 10.1.15.* 10.1.19.*
10.2.10.* 10.2.11.*
192.168.22.* 192.168.24.* 192.168.25.*
Destination address must be in one of the same following regex ranges.
Service (destination port) must be one of:
Exactly port: 1310, 1411, 1812, or any port starting with 455
No protocol is specified, so it will match either TCP or UDP

fw1logsearch.pl will output any matching events to stdout, including a FW1 log header line, so the next instance of fw1logsearch.pl continues filtering the result set.

3. The second fw1logsearch.pl specifies Source Address must not be any of the following
192.168.22.14

10.2.11.12

4. The last fw1logsearch.pl excludes port 1310 from 192.168.24.12, and puts all those records into a separate reject file, while writing the other records to stdout.

This script has been used to process over 4 billion records within the project I wrote it for – and precisely found all the use of particular business cases I needed to modify.  The result was zero outages and no unintended business interruption.

Basic syntax/help file:

Usage:  fw1logsearch.pl
[-a|--incaction|-A|--excaction <action regex>]
[-p|--incservice|-P|--excservice <dst port regex>]
[-b|--incs_port|-B|--excs_port <src port regex>]
[-s|--incsrc|-S|--excsrc <src regex>]
[-d|--incdst|-D|--excdst <dst regex>]
[-o|--incorig|-O|--excorig <fw regex>]
[-r|--incrule|-R|--excrule <rule-number regex>]
[-t|--incproto|-T|--excproto <proto regex>]

[--dnscache <dns-cache-file>]
[--resolveip]
[--allinclude]
[--allexclude]
[--rejectfile <file>]
[--debug <level>]

fw1logsearch.pl will search a fwm logexport text file for regex patterns specified for supported columns (such as service, src, dst, rule, action, proto and orig).

Include and exclude regex matches may be specified on the same line, although they both will include (print) a line or exclude (reject) a line based on single matches.  Allinclude or Allexclude must be specified to force a match
only on all specified column regex patterns.

Regex patterns can be enclosed with single quotes to include characters that are special to the shell, such as the ‘or’ (|) operator.

Header will be output only if there are any matching lines.

Example invocations:
$ cat 2008-07-07*txt | \
fw1logsearch.pl \
-p ‘53|domain’ \
-d ‘192.168.1.2|host1|10.10.1.2|host2′ \
-o ‘192.168.2.3|10.10.2.4|10.10.4.5′ \
-S ‘64.65.66.67|32.33.34.35|10.10.*|192.168.*’ \
–resolveip
Will require destination port (service) to be 53, destination IP to be any of 192.168.1.2, host1, 10.10.1.2, or host2  the reporting firewall (origin) to be any of 192.168.2.3, 10.10.2.4, or 10.10.4.5  and the source IP must not be
any of 64.65.66.67, 32.33.34.35, 10.10.*, or 192.168.*  Any lines that match this criteria, will display and the orig, src, and dst columns will use the default DNS cache file (dynamically built/managed) to perform name resolution, replacing the IP addresses where possible.

Include regex patterns:
-a  –incaction    Rule action (accept, deny)
-b  –incs_port    Source port (s_port)
-p  –incservice   Destination port (service)
-s  –incsrc       Source IP|hostname
-d  –incdst       Destination IP|hostname
-o  –incorig      Reporting FW IP|hostname
-r  –incrule      Rule number that triggered entry
-t  –incproto     Protocol of connection

Exclude regex patterns:
-A  –excaction    Rule action (accept, deny)
-B  –excs_port    Source port (s_port)
-P  –excservice   Destination port (service)
-S  –excsrc       Source IP|hostname
-D  –excdst       Destination IP|hostname
-O  –excorig      Reporting FW IP|hostname
-R  –excrule      Rule number that triggered entry
-T  –excproto     Protocol of connection

Other options:
–debug {level} Turn on debugging
–dnscache      Specify location of DNS cache file to be used with
the Resolve IPs option
–resolveip     Resolve IPs for orig, src, and dst columns AFTER filtering
–rejectfile    Write out all rejected lines to a specified file

Download fw1logsearch.pl

Encrypted Filesystems with Sparse Bundles
Mac OS X offers encrypted filesystems through sparse bundles.  To mount up a sparse bundle, given the password used to create the bundle, use the hdiutil:

hdiutil attach -verbose -readonly /path/to/sparse.bundle.directory

This will mount up the sparse bundle located at the directory path specified.  To unmount the sparse bundle, use:

hdiutil detach /Volume/sparse.bundle.name

Adding entries to /etc/hosts
Although simply editing /etc/hosts should work, there are times where the new entries may not be recognized, in these cases the OS X name cache daemon needs to be kicked:

dscacheutil -flushcache

Mac OS X Hostnames
Although you can change the hostname of your Mac OS X device through the System Control Panel -> Sharing, the following command line can lock the name so DHCP and other dynamic networking protocols don’t mess up your hostname (from RichardBronosky):

sudo hostname my-permanent-name

sudo scutil –set LocalHostName $(hostname)

sudo scutil –set HostName $(hostname)

Fast way to synchronize the content of your iTunes libraries – this doesn’t sync the playlists or any iTunes meta information (and you may need to perform an Add to Library .. to import any new content). This was just a quick and dirty way to sync up my iTunes downloads with another iTunes library at home. This assumes that you’ve opened up the ability to Remote Login (ssh) to the target Mac (topic for another time).

rsync -av -e ssh "Music/iTunes/iTunes Music/" ahull@10.20.1.103:"/Users/ahull/Music/iTunes/iTunes\ Music"