Skip to content

Setting up VLAN tagging on an OpenWRT router (My Net N750)

14-Aug-14

I have pfSense box running with an Intel PCI dual-port gigabit NIC.

However, I’ve recently discovered that this PCI card draws around 10 watts. (I unplugged the PCI card and measured on my Kill-A-Watt.)

So, I came up with a different idea: use an OpenWRT device ahead of the pfsense box to VLAN-tag WAN vs LAN traffic. This additional device allows me to use only one NIC (the motherboard NIC) on my pfsense box. I can remove the PCI card, and all will be well.

It turns out that setting this up was easier to do than I thought. (I was also contemplating buying a managed switch.)

I flashed the OpenWRT image using instructions here.  I used the -RC1 image first, and then re-did this with trunk. To be honest, I wasn’t sure what I was doing the first time, so I’m not sure if the version matters.

I logged in using telnet and set a password.

I then logged in using ssh (just for kicks) and began editing configuration, using the Dumb AP instructions. The only difference is that I didn’t remove the VLAN for the WLAN–instead, I just left it and I made port 4 tagged for both VLAN & WLAN. Here is my /etc/config/network; note that both switch_vlan have a 4t in them–port 4 is a member of both VLANs (and is tagged):


root@OpenWrt:~# cat /etc/config/network

config interface ‘loopback’
option ifname ‘lo’
option proto ‘static’
option ipaddr ‘127.0.0.1’
option netmask ‘255.0.0.0’

config globals ‘globals’
option ula_prefix ‘fd9f:d820:9e00::/48’

config interface ‘lan’
option ifname ‘eth0.1’
option force_link ‘1’
option type ‘bridge’
option proto ‘static’
option ipaddr ‘192.168.1.5’
option netmask ‘255.255.255.0’
option ip6assign ’60’

# config interface ‘wan’
# option ifname ‘eth0.2’
# option proto ‘dhcp’
# option macaddr ’00:90:a9:cd:a8:40′

#config interface ‘wan6’
# option ifname ‘@wan’
# option proto ‘dhcpv6’

config switch
option name ‘switch0’
option reset ‘1’
option enable_vlan ‘1’

config switch_vlan
option device ‘switch0’
option vlan ‘1’
option ports ‘0t 1 2 3 4t’

config switch_vlan
option device ‘switch0’
option vlan ‘2’
option ports ‘0t 4t 5’

Also, the instructions say how you can disable dnsmasq on one interface only, but what you really want to do (probably) is:


root@OpenWrt:~# /etc/init.d/dnsmasq disable

I then also disabled firewall, and reloaded network settings. I haven’t installed LUCI (it appears that it does not come with the N750 images), and that’s fine–I probably won’t use it much.

Be the first to like.

I get my wish (a pfSense router)

13-Aug-14

Remember back when I was considering a pfSense build? Well, it turns out it was a lot easier than I though.

I realized I could buy (for about $7) an Intel dual-PCI Ethernet card (HP NC7170), and the $6 riser card would allow me to fit it into my case. (I had to do a little bit of metal cutting with a Dremel, since there are no PCI slots in the back of the computer.)

So, that allowed me to get full gigabit speeds off the pfSense. In addition, I’m now* using Intel cards to do so, so that should be better in terms of offloading the CPU.

Here is a picture of this build:

pfSense build with Intel NIC wedged in

pfSense build with Intel NIC wedged in

Note that the Intel NIC is sort of resting on the ac/dc adapter (which I also stuffed into the case, since I had removed the PSU that came with the case). I later reinforced it (sort of) with zip-ties.

This is something I meant to post earlier, but forgot for a long time. I finally have a free night (where I’m not working), so I thought I’d catch up.

* And since I built this, I’ve changed my setup. I noticed that the PCI card was drawing about 10 watts—meaning, when it was plugged in, I’d see 35 Watts total, and when I removed it, it was 25 watts total. (This is steady-state power, after the router booted up.) So, I went to vlan tagging with a openWRT router, and am using the single gigabit Ethernet that is built-in to the motherboard. At first, I worried about using a Realtek NIC, but I haven’t had a single problem with them. The point with the Intel cards was to offload the CPU, but the higher power doesn’t seem worth it. Finally, I notice that the Alix boards all have Realtek NIC’s.

Be the first to like.

Arch Linux ARM on a Pogoplug Series 4

02-Jul-14

I followed the directions here: Pogoplug Series 4 | Arch Linux ARM.

It took only about 15 minutes, and the iperf scores are outstanding:

The install was on an 2.5″ SSD disk I had laying around, which explains the fast install. But that’s the point–the pogoplug 4 has a SATA port, so I can use an SSD.

Be the first to like.

Making duplicity and ncftpput play nice with vsftpd FTP daemon

25-Jun-14

I’ve been setting up yet another remote backup lately (see associated problem here). For this purpose (on Unix), the duplicity solution looks ideal. However, I’ve tried it on a couple of lightweight FTP servers (a TL-WDR3600 and a Raspberry Pi) and neither of them work. I keep getting a “Permission Denied” message.

I did quite a bit of investigation, and found that it’s related to the way that duplicity’s FTP backend, ncftp/ncftpput sends files. Or really, a quirk of vsftpd, where it won’t let you put a file with an absolute path (and more specifically a file with a directory name). I’ve seen other reports of this on the Internet here and here.

Here’s a sample ncftpput debug log:

2014-06-25 00:56:34  331: Please specify the password.
2014-06-25 00:56:34  Cmd: PASS xxxxxxxx
2014-06-25 00:56:34  230: Login successful.
2014-06-25 00:56:34  Cmd: PWD
2014-06-25 00:56:34  257: "/"
2014-06-25 00:56:34  Logged in to 192.168.1.2 as Poojan.
2014-06-25 00:56:34  Cmd: FEAT
2014-06-25 00:56:34  211: Features:
2014-06-25 00:56:34        EPRT
2014-06-25 00:56:34        EPSV
2014-06-25 00:56:34        MDTM
2014-06-25 00:56:34        PASV
2014-06-25 00:56:34        REST STREAM
2014-06-25 00:56:34        SIZE
2014-06-25 00:56:34        TVFS
2014-06-25 00:56:34        UTF8
2014-06-25 00:56:34       End
2014-06-25 00:56:34  Cmd: PWD
2014-06-25 00:56:34  257: "/"
2014-06-25 00:56:34  Cmd: CWD folder1/duplicity/Public
2014-06-25 00:56:34  250: Directory successfully changed.
2014-06-25 00:56:34  Cmd: CWD /
2014-06-25 00:56:34  250: Directory successfully changed.
2014-06-25 00:56:34  Cmd: TYPE I
2014-06-25 00:56:34  200: Switching to Binary mode.
2014-06-25 00:56:34  Cmd: PASV
2014-06-25 00:56:34  227: Entering Passive Mode (192,168,1,2,225,89).
2014-06-25 00:56:34  Cmd: STOR folder1/duplicity/Public/duplicity-full.20140625T055614Z.vol1.difftar.gpg
2014-06-25 00:56:34  550: Permission denied.
2014-06-25 00:56:34  ncftpput folder1/duplicity/Public/duplicity-full.20140625T055614Z.vol1.difftar.gpg: server said: Permission denied.
2014-06-25 00:56:34  Cmd: QUIT
2014-06-25 00:56:34  221: Goodbye.

However, I ran a few tests, and I found that if one cd‘es to the directory and then puts (STOR‘es) the file, everything works. My guess this is a chroot-style feature. So, I created a small Python script that will parse the arguments that duplicity sends, and inserts cd (to the target directory) command before uploading the file. This works well. Here’s the script:


#!/usr/local/bin/python3
import argparse
import subprocess
import sys
import os

# ncftpput looks something like this:
# 'ncftpput -f /tmp/duplicity-yRftm3-tempdir/mkstemp-PiYFTL-1 -F -t 30 -o useCLNT=0,useHELP_SITE=0  -m -V -C '/tmp/duplicity-yRftm3-tempdir/mktemp-sWc1AA-3' 'folder1/duplicity/Public/duplicity-full.20140625T041147Z.vol1.difftar.gz
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='wrapper for ncftpput command, to allow absolute target paths on vsftpd')
parser.add_argument('-f', dest='credentials_file')
parser.add_argument('-F', action='store_true')
parser.add_argument('-t', dest='timeout')
parser.add_argument('-o', dest='ftp_options')
parser.add_argument('-m', action='store_true')
parser.add_argument('-C', action='store_true')
parser.add_argument('-V', action='store_true')
parser.add_argument('source')
parser.add_argument('dest')
args = parser.parse_args()

# print(args)

dest_dir = os.path.dirname(args.dest)
dest_file = os.path.basename(args.dest)

#cmd_args = ['/usr/local/bin/ncftpput', '-d', 'ftp-debug.log', '-W', 'cd {0}'.format(dest_dir)] +  sys.argv[1:-1] + [dest_file]
cmd_args = ['/usr/local/bin/ncftpput', '-W', 'cd {0}'.format(dest_dir)] +  sys.argv[1:-1] + [dest_file]
subprocess.Popen(cmd_args)

The script is called ncftpput, and is placed in a directory called ~/duplicity_bin. I then add this directory to my path before running duplicity. Here’s a shell script that does that:


#!/bin/sh
export FTP_PASSWORD="XXXYYYXXYXYZY"
export PATH="/home/Poojan/duplicity_bin:$PATH"
duplicity -v 9 --encrypt-key=DEADBEEF full /tank/Users/Public ftp://Poojan@192.168.1.2/folder1/duplicity/Public

Be the first to like.

Keeping FreeBSD TCP performance in the midst of a highly-buffered connection

21-Jun-14

I was perplexed recently, when I began an rsync job to a raspberry pi server. I know exactly what limits the bandwidth of this connection–it is the CPU (or network) on the Raspberry Pi, which cannot accept data fast enough.

So, even though my server is on a 1 Gbit/s interface, and the Raspberry Pi is on a 100 Mbit/s interface, the transfer rate is ~ 10 Mbit/s. Fair enough.

But what really perplexed me is that the presence of this rsync connection severly limited other connections–notably Samba. The Simpson’s show in the living room had audio that was noticeably stuttering.

So, I began to investigate. This same low-rate occurred with iperf. It seemed a little better from my basement computer than the living room machine. Here is an iperf from the basement to the FreeBSD server:


C:\Users\Poojan\Downloads\iperf-2.0.5-2-win32>iperf -c server
------------------------------------------------------------
Client connecting to server, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.20 port 64155 connected with 192.168.1.8 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 25.1 MBytes 21.0 Mbits/sec

Whereas without rsync going, it would be around 670 Mbit/s or so.

I started playing around with buffers. Curiously, reducing sendbuf_max helped:


Poojan@server ~ >sudo sysctl net.inet.tcp.sendbuf_max
net.inet.tcp.sendbuf_max: 262144
Poojan@server ~ >sudo sysctl net.inet.tcp.sendbuf_max=65536
net.inet.tcp.sendbuf_max: 262144 -> 65536

Which yielded:

C:\Users\Poojan\Downloads\iperf-2.0.5-2-win32>iperf -c server
------------------------------------------------------------
Client connecting to server, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.20 port 64171 connected with 192.168.1.8 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 788 MBytes 661 Mbits/sec

I posited that maybe there’s some overall limit to the buffers, and rsync was stealing all of them, so making them smaller allowed more buffers to be available to iperf. I went hunting for this limit.

I tried doubling kern.ipc.maxsockbuf:


Poojan@server ~ >sudo sysctl -w kern.ipc.maxsockbuf=524288
kern.ipc.maxsockbuf: 262144 -> 524288

which yielded:

C:\Users\Poojan\Downloads\iperf-2.0.5-2-win32>iperf -c server
------------------------------------------------------------
Client connecting to server, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.20 port 64216 connected with 192.168.1.8 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 25.0 MBytes 20.9 Mbits/sec

No luck. Note: I realized that the above was with Jumbo frames enabled on both server & client. I disabled jumbo on client.

I then did a netstat -m, just in case:


1470/5175/6645 mbufs in use (current/cache/total)
271/2635/2906/10485760 mbuf clusters in use (current/cache/total/max)
271/2635 mbuf+clusters out of packet secondary zone in use (current/cache)
85/335/420/762208 4k (page size) jumbo clusters in use (current/cache/total/max)
1041/361/1402/225839 9k jumbo clusters in use (current/cache/total/max)
0/0/0/127034 16k jumbo clusters in use (current/cache/total/max)
10618K/11152K/21771K bytes allocated to network (current/cache/total)
1106/2171/531 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
361/1345/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

This didn’t really show any indication that buffers were being over-subscribed, at least not during the tests.

But now, with a sendbuf_max size of 262144, and a maxsockbuf size of 524288, my iperf reading went down:


C:\Users\Poojan\Downloads\iperf-2.0.5-2-win32>iperf -c server
------------------------------------------------------------
Client connecting to server, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.20 port 64438 connected with 192.168.1.8 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.7 sec 2.88 MBytes 2.25 Mbits/sec

From reading this summary of FreeBSD buffers, it seems that kern.ipc.maxsockbuf operates at a different level than net.inet.tcp.sendbuf. And, in fact, both these being large is impacting the performance. So, maybe this is just pure buffer bloat.

But, then I realized that my better results were when the sendbuf was less than 64k. So, I disabled RFC1323 (which allows for buffers larger than 64k, in addition to time-stamps). And voila!


C:\Users\Poojan\Downloads\iperf-2.0.5-2-win32>iperf -c server
------------------------------------------------------------
Client connecting to server, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.20 port 65203 connected with 192.168.1.8 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 798 MBytes 669 Mbits/sec

Be the first to like.

worrbase – CrashPlan on FreeBSD 9.0, A HOWTO

15-Jun-14

I’m blogging this as a reminder to myself on what to do. I found it years ago, but that link doesn’t seem to exist anymore. Anyway, this is a good write-up on FreeBSD 9.0:

worrbase – CrashPlan on FreeBSD 9.0, A HOWTO.

The only difference betwen this and what I did was that I use Oracle JRE 1.8 /usr/ports/java/linux-oracle-jre18/, so my JAVACOMMON looks like:

JAVACOMMON=/usr/local/linux-oracle-jre1.8.0/bin/java

Be the first to like.

NFSv4 ACL history

28-May-14

Good summary of NFSv4 ACL’s (and their history):

Implementing Native NFSv4 ACLs in Linux (by Greg Banks at SGI).

Be the first to like.

Crucial m500 120GB as a ZFS slog

12-Apr-14

I re-did my benchmarks of my ZFS ZIL using a Crucial m500. The main reason I got this drive is that it has power-loss protection. And, it’s speeds are adequate to keep up with my gigabit Ethernet samba server.

Here are the results. My first run was decent, but not stellar:

I didn’t do a [ccie]zpool iostat[/ccie] during this run. This matches around what I was getting as a 4k random write. My second run was much better:

And it matches a typical zpool iostat:

Overall, I’m pretty happy. I was hoping to get at minimum 125 MB/s, which my gigabit Ethernet can’t really saturate.

I can now even try turning on forced sync writes in Samba.

Be the first to like.

Crucial m500 120GB benchmarks

09-Apr-14

The latest in my obsession with SSD’s. I jumped on a $70 deal at NewEgg. I bought it to use as a SLOG (ZIL) in my ZFS server, because of its write speeds and because of its power loss protection.

I immediately updated the MU03 firmware to MU05.

This may be with a SATA II (3 Gbps) cable:

Wha? The 4K random write is quite low. Let’s repeat:

Little better at 76 MB/s random 4k write, but not the 120 MB/s that I expected.

Let’s try the native SATA III port and a shorter cable:

Wow! that sequential read is fast. But, the random write (QD=1) is still disappointing. What gives?

Post Script

Looks like someone else gets similar results, which don’t line up with what TweakTown measured.

Oh, well. The power loss protection is important, and it really only matters how the drive performs in the ZIL. Hopefully, I’ll see closer to 140MB/s rather than 65 MB/s.

Update 2014-04-09

Tried re-running with Intel’s RST driver:

Slightly better, but still no 120 MB/s. I have a feeling that something changed with the latest firmware. Wish I had run this before I upgraded.

Be the first to like.

More ZIL/SLOG comparisons

05-Apr-14

Suspicious of my previous tests using iozone, I wrote a small program (that follows) to measure synchronous write speed. I tested this with and without an SLOG device, and then used a really fast Plextor M5 Pro 256GB SSD as the SLOG. I wanted to make sure that something could achieve the fast SLOG speed that I hoped for. (Remember I am doing this in the interest of spending money on a new SSD.)

Summary

An SLOG device can degrade performance, by forcing synchronous writes to wait until they are committed to the SLOG. However, it cannot enhance performance. The max (steady-state) performance is determined by bandwidth to the main storage.
More…

Be the first to like.