Saturday, October 1, 2011

Force downloading through dynamic Content-Disposition

The Problem

I wanted to make it to images on my personal site could be downloaded easily. But I also want them to be normally visible in the browser. I decided to solve this with a query string parameter so:
http://epii.info/anime/2011/Fanime/Ace%20Attorney%20-%20Trucy%20Wright%208.jpg
will load the image (of Trucy from Ace Attorney), and
http://epii.info/anime/2011/Fanime/Ace%20Attorney%20-%20Trucy%20Wright%208.jpg?dl
will give the user the option to download it.

Basically I want to conditionally apply a Content-Disposition: attachment header to essentially static files, depending on the query-string.

The Solution

I found a close, but non-working example at Experts Exchange. I revised that into this:
RewriteCond %{QUERY_STRING} ^dl
RewriteRule ^((.*/)*([^/]+jpg))$ - [E=dlfilename:$3,E=fullpath:$1]
Header onsuccess set Content-Disposition "attachment;filename=\"%{dlfilename}e\"" env=dlfilename
Header onsuccess set Content-Location "/%{fullpath}e" env=dlfilename
Header onsuccess set Link "</%{fullpath}e>; rel=\"canonical\"" env=dlfilename

which is placed in an Apache 2.2 .htaccess file in the shared-hosting webserver's root.
It assumes mod_rewrite is installed and turned on (via RewriteEngine on), as well as mod_headers

Breakdown

RewriteCond %{QUERY_STRING} ^dl
The RewriteRule now only applies if the query string starts with "dl"

RewriteRule ^((.*/)*([^/]+jpg))$ - [E=dlfilename:$3,E=fullpath:$1]
This doesn't rewrite anything, but rather assigns two Apache environment variables if the file ends in jpg (and the previous RewriteCond passed). dlfilename is set with just the file name, not the full path. fullpath contains the full path (relative to .htaccess) to the requested file.

Header onsuccess set Content-Disposition "attachment;filename=\"%{dlfilename}e\"" env=dlfilename
If the dlfilename environment variable has been set, this sets the Content-Type to suggest the user saves the file, and sets the filename to the file's name. (could be problematic as implemented here if your file names contain quotes.)

Header onsuccess set Content-Location "/%{fullpath}e" env=dlfilename
This sets the Content-Location to offer a canonical location of the file. The idea is to improve image SEO by telling the search engine they aren't different. Google says they ignore this header due to it being popularly misconfigured. Just in case anyone does use it though, it's there.

Header onsuccess set Link "</%{fullpath}e>; rel=\"canonical\"" env=dlfilename
More SEO. This is Google's suggested alternative to Content-Location, though they say it only applied to Web Search for now. So if you're using images like me, this may not help you any. But if you're using PDFs, it probably will.

Saturday, July 23, 2011

FreeBSD ipfw, nat,IPsec, and IPv6 DNS

I recently setup a FreeBSD 8.3 (amd64) system as a router for a local network. The internal network is dual stack IPv4 + IPv6. This required a custom configured firewall as well beyond the standard "simple", "open", etc.

ipfw + natd

The first unexpected event happened when discovered that after rebooting, NAT no longer worked. Some investigating revealed that when processing firewall rules, when adding
${fwcmd} add 150 divert natd all from any to any via ${public_if}

during boot, it failed with the error
ipfw: getsockopt(IP_FW_ADD): Invalid Argument

Apparently I've become too expectant of relevant kernel modules to just be there. The ipdivert module wasn't being loaded until natd needed it, which is after ipfw initialized.

Solution: Add ipdivert_load="YES" to /boot/loader.conf - then it's ready well before ipfw needs it.


IPsec

In general, I've avoided using IPsec for VPNs in favor of OpenVPN or tinc. But since much of the world uses IPsec, I figured I should get more experience with it. (After this experience, I still strongly lean toward those.) I followed the guide for FreeBSD VPN over IPsec. That worked for a FreeBSD 7 system, but I ran into a non-fatal problem on my FreeBSD 8 box:
WARNING: setsockopt(UDP_ENCAP_ESPINUDP_NON_IKE): UDP_ENCAP Invalid argument

Solution:kernel config needs IPSEC_NAT_T enabled as well as IPSEC:
options IPSEC #IP security
options IPSEC_NAT_T
device crypto

(I found it surprising this wasn't all part of GENERIC by default - seems most features are, and those that aren't are in a loadable kernel module)

I fan into further problems with SA negotiation, due to not quite properly swapping EVERY IP in the config between the two machines.

I think it's worth not note to anyone reading this: make sure you test with a packet sniffer (tcpdump or wireshark or similar) - even if the secure link is not setup, packets will still transmit, just in the clear. So just because the link works doesn't mean the "private" part of the VPN is there.


DNS & IPv6 bootstrapping

More for the hell of it than any legitimate reason, I wanted to use the SixXS IPv6 DNS servers to access Google's sites via IPv6. Including from the machine working as a router. But that presented a few problems:

dhclient insists on updating resolv.conf

I need a DHCP client to get the dynamic IP for the public interface. Putting anything in /etc/resolv.conf gets overwritten. Also, IPv6 addresses are not accepted in /etc/dhclient.conf. So, I ran a nameserver on the localhost, listening on IPv4 and pointed /etc/dhclient.conf to it:
supersede domain-name-servers 127.0.0.1;

PowerDNS and IPv6

With its scary security history, and repeated new vulnerabilities, I do not run BIND. My preference has been DJBDNS tinydns/dnscache. But those lack IPv6 support. So I chose PowerDNS instead.
Doing an initial test of dig @127.0.0.1 aaaa ds.v6ns.test-ipv6.com did not work, and I discovered that unless you setup something for the outgoing IPv6 address in /usr/local/etc/pdns/recursor.conf, it won't attempt to use IPv6 while resolving DNS:
#send IPv6 queries using any source
#"::" is similar to "0.0.0.0" for IPv4
query-local-address6=::

sixxs-aiccu needs hostnames to connect

With that working, I decided to make it pass on all requests to SixXS by adding
forward-zones-recurse= .=2001:4de0:1000:a3::2, .=2001:4de0:1000:a4::2
to /usr/local/etc/pdns/recursor.conf
However on the next reboot, sixxs-aiccu wouldn't start. As I had it configured, it was connecting by hostname. So it needs working DNS before making the bridge to the IPv6 internet.

I could alter it to connect by IP, but who knows what other services may have similar problems. Since I was many interested in using it for accessing Google sites, I just changed my /usr/local/etc/pdns/recursor.conf to use:
forward-zones-recurse= google.com=2001:4de0:1000:a3::2, google.com=2001:4de0:1000:a4::2

which only uses SixXS for Google related requests, an IPv4 root servers by default, allowing it to look up the names of the servers needed to start the IPv6 tunnel through SixXS.

(Side note: youtube.com and blogger.com are CNAMEs for google.com addresses, so the AAAA lookup still goes to the SixXS servers for them, and I'm now posting here via IPv6)

Sunday, February 28, 2010

outboundlink.net / DrivingRevenue

I noticed some hyperlinks redirecting via outboundlink.net
At first I was paranoid about spyware, but after doing some research, I discovered it's related to DrivingRevenue - which converts hyperlinks on some social sites into affiliate marketing links. This makes it so the site owner gets paid if users followed links from their site to the linked site and buys something.

On livejournal.com, it seems to be dRev.js that takes care of the link rewriting.

Saturday, October 24, 2009

simple cometd / jetty private client messaging

Recently I've been researching comet servers and protocols. Based primarily on the Comet Daily server maturity guide / feature matrix, I was split between Jetty/java-cometd & Orbited. Since Orbited's released version did not support cross-domain connections, I went with Jetty/java-cometd.

I needed:
  • Cross-browser support
  • No need for broadcast messages
  • Ability to send messages to specific clients
The application I am writing for is not Java based. At the moment Java does not play any role in our environment. What extremely limited documentation there already is seems to assume familiarity with Java servlets and Maven. I have no prior Maven experience, and the last time I seriously wrote any Java code was 2002. This will hopefully help others in the same situation. I don't claim this to be "right" just that it works for me.

Saturday, September 26, 2009

piclenslite vs Firefox 3 XHTML

A website of mine (http://epii.info/anime/) is using the piclenslite script to facilitate image browsing. It stopped working in Firefox, and after some frustration tracking down the problem, it seems this is due to my site being XHTML. When served up with Content-Type: application/xhtml+xml; charset=UTF-8, the script does not work, but when served up as Content-Type: text/html; charset=UTF-8, it does work.

Monday, August 31, 2009

FreeBSD on Sony Vaio TZ (VGN-TZ195N)

I bought a new to to replace by Fujitsu T70s. It was among very few options offering the minimal computing power I wanted in a sufficiently compact size. Vaio TT is more of what I want, but our of my current price range.

Round 1 - PC-BSD + Vista
Not realizing Vista's booting would work different than XP, I took the usual install steps:
  • Shrink NTFS partition by booting Knoppix & using QTPartEd
  • Install PC-BSD with the default boot manager
PC-BSD booted well out of the box. X worked (with proper resolution easily set). Audio. No WiFi. Rebooted to check out a test file I made in the NTFS partition.

Vista won't boot. Search for insight. Apparently Vista doesn't take kindly to setting up your own MBR. It seems based on an article on installing FreeBSD on a Vista box that you can make it work, or use Vista's boot loader to dual-boot FreeBSD. Unfortunately, I found those after some fussing which messed up by recovery partition, and ultimately killing any chance of getting Vista working on there again without an install CD. So, time for BSD-only. Probably the better choice, because Vista demanding 25GB of my 48GB SSD disk didn't leave much to spare anyway. (I just wanted to keep it because sometimes having Windows is handy for trying out random Windows-only stuff.)

Round 2 - FreeBSD 8-Beta
Some researh on my wireless card indicates it needs the the iwn(4) driver that is not part of FreeBSD 7, but in part of 8. At the time of writing FreeBSD 8 is in beta. I wanted to try out PC-BSD, but not now that will wait.

Got the boot-only CD for the amd64 architecture. Installed via FTP.

Wireless
My NIC needs iwn. Didn't detect the NIC out of the box. Seems loading the kernel module (i.e. if_iwn_load="YES" in /boot/loader.conf) would have sufficed, but since I wanted to switch off some of the debugging options in favor of performance I created a new kernel config and added:
device iwn
Upon reboot, the NIC showed up. Trying to get an access point list via
ifconfig iwn0 up list scan
Failed. "unable to get scan results" Some research uncovered that now the correct way to manage wireless NICs in FreeBSD is not directly, but though a wlan device, requiring something like:
wlans_iwn0="wlan0"
ifconfig_wlan0="ssid StationName nwkey ****** DHCP"
in rc.conf. (Or manual creation of the wlan0 interface.)

Ports
I installed some packages from sysinstall. They didn't work with errors about missing libraries. Seems they where maybe looking for FreeBSD 7 ones? Symlinking the requested version to the installed version seemed to be working, but isn't really a proper solution. On well. Off to ports instead of packages.

Build Tweaks
Since I'm compiling from source, I added the following to /etc/make.conf (maybe not as important for amd64 as i386...)
CPUTYPE=core2
BUILD_OPTIMIZED=Y
CFLAGS=-O2 -pipe -fno-strict-aliasing
Xorg
By default, xorg wanted to run at 1024x768, not 1366x768. Solution was adding
Option      "NoDDC" "Yes"
to the Device Section, and
    SubSection     "Display"
Depth 24
Modes "1366x768"
EndSubSection
to the Screen section.

KDE4
kded4 (via kdeinit4) eats up lots of CPU. Annoying on my desktop, and a battery life concern on my laptop. Apparently this is because it polls files for updates rather than using gamin+kqueue.
http://lists.freebsd.org/pipermail/freebsd-ports/2008-October/050961.html

KDE4 also seems to eat up ~300MB ram compared to just a simple window manager. I like the KDE Desktop overall, but for RAM intensive work like photo management with Bibble, I may need to go with something lighter.

Flash/Java Plugins
Being stuck with Flash 7 in FreeBSD has been frustrating. Getting Flash 10 working in FreeBSD 8 required a little bit of poking around to get working. I followed the instrustions on installing Flash & Java browser plugins from the FreeBSD Handbook. But had no working flash or Java. I ultimately found this post, and discovered there was an extra step needed:
cd /usr/local/lib/firefox3/plugins
ln -s /usr/local/lib/browser_plugins/npwrapper.libflashplayer.so .
ln -s /usr/local/diablo-jre1.6.0/plugin/amd64/ns7/libjavaplugin_oji.so .

Suspend/Resume
S3 (keep RAM/suspend) works sometimes, however sysmouse needs to be restarted.
S4 (to disk/hibernate) does not work


Battery Life
I did a test seeing how long battery lasted under fairly light use, with wireless enabled. It lasted 4h 45m

I noticed acpiconf tells me the power usage, So I did some tests to see what saved how much power compared to running with the lowest LCD brightness at 100Mz (idle) with wireless ON.
FeaturePower
Base11650mW
LCD OFF-100mW
LCD MAX Brightness+2700mW
Wireless OFF-1700mW
1200MHz CPU (idle)+50mW(?)
hw.acpi.cpu.cx_lowest=C3-2300mW
Power Down internal USB devices via acpiconf-600mW
Power Down internal USB devices except DVD-500mW
I'm surprised that the CPU frequency change seems to have such a small impact on power usage. The CX State does, but at C3 sometimes the system gets kinda choppy in its responsiveness. Setting one core to C2 looses about half the power savings but gets rid the the choppiness.

With much tweaking (low LCD, usb devices off, wireless off, cpu settings, etc.) I got it down to 6743mW, which should be a little over 9hrs. But I doubt I'll be using it for 9hr marathons of editing text files in the dark, so that metric is useless.

Since the internal USB devices except the CD-ROM drive have no BSD drivers, and even if they did, I doubt I'd use them, I setup an rc script to power them off on boot.
#!/bin/sh
#
# PROVIDE: save_power
# REQUIRE: devd
# KEYWORD: nojail
#

. /etc/rc.subr

name=save_power
rcvar=`set_rcvar`

start_cmd=${name}_start

save_power_start()
{
UC=/usr/sbin/usbconfig

${UC} -u 3 -a 2 power_off #Fingerprint Sensor
${UC} -u 4 -a 7 power_off
${UC} -u 4 -a 6 power_off
${UC} -u 4 -a 5 power_off
${UC} -u 4 -a 4 power_off #Bluetooth
${UC} -u 4 -a 3 power_off
${UC} -u 4 -a 2 power_save #DVD

sysctl dev.acpi_sony.0.brightness=3 #start not too bright
}

load_rc_config ${name}
: ${save_power_enable=YES}

run_rc_command "$1"
As a side note, Sony seems to think the computer can get 4-8hrs on battery,

Other references:

Thursday, August 27, 2009

Wireless Networking in Vista

I just started using Vista. It's not quite as unmanageable as I expected, but perhaps that's because I haven't used Windows seriously in a long time.

My first challenge that did involve wasting quite a bit of time to figure out was getting wireless networking to work. The solution was simple, but non-obvious:

  • I guess for battery saving purposes, my Sony TZ had Wireless disabled, and WWAN (cellular) enabled. I needed to use the special Sony SmartWi to tell it to turn that off and turn on 802.11 Wireless.
  • Our router is set to not broadcast its SSID. So, I went to add the unlisted network, but it wouldn't connect. It wouldn't even list it as an option to connect to. I found to actually see it, I needed to go to: Network part of Control Panel. Then go to manage wireless from there.
  • But that still wouldn't connect. To make it actually connect, I had to go to its properties and select the option to "connect even if the network is not broadcasting".
More detail is available in a Microsoft TechNet Article on non-broadcasting wireless access points.