Author: frankd412

Me vs my 2006 Audi A4 Quattro 2.0T Part I

The last month has been trying with this vehicle.. I hated Audis earlier in life, to which most people for some reason thought I was jealous. I wasn’t. There was a reason.

And this is the month that I’m reminded of it.

Quick background: I bought the car about 10 months ago and have had no problems with it aside from a miniscule coolant leak and some squealing from the turbo. Those things I was willing to put up with until it was warmer out. I ordered an S3 turbo which is a K04, I believe slightly smaller than the one on my 2007 Solstice GXP — but much bigger than the stock K03 and the same price as the puny stock one. And who could argue with more power, right? And I really don’t mind working on my own cars so long as I’m not fixing them. If I’m UPGRADING them, that’s a different story, right? So it got warm out.

So it all started when I went to put UV dye in my coolant system to track down this little leak that left no puddles or obvious signs, it was extremely slow. I hop in the car and go to drive it to get it good and hot and try to see if I can get my UV light and find some signs of the leak. Can you guess what happened within the next 15 minutes?

The car overheated. Quickly. I put the heat on, and that wasn’t working which told me I had just lost a massive amount of water. I was very close to home and decided to risk the 1/4 mile trip with as little throttle as possible. Audi likes to do this cool thing where they make everything out of plastic. Like coolant flanges on that bolt on to the back of the engine in a nigh-unreachable spot and connect the motor’s coolant passages to the heater hose and upper radiator hose and god knows what else.

Plastic is awesome.

Plastic is awesome.

 

This little piece of plastic took hours. And they break. All the time. Awesome design, Audi! It’s a cheap part, I think I paid around $20. Then I paid $100 for a new heater hose that connects to it just for peace of mind, and about $15 on another hose that’s about 4″ long (it’s probably one of the more expensive 4″ pieces of black hose you’ll run into).

So I somehow get the coolant flange in there and manage to get the hoses on it, I bolt the cam-driven vacuum pump back on and the high pressure fuel pump — I buy another emissions related hose and that sets me back $75 as the clips on mine broke. The PCV valve which is plastic, of course (and no it’s not a Positive Crankcase pressure Valve, it’s Audi’s ridiculous idea of Crankcase regulation) has nice hot oil fumes coming up into it — which make the plastic very brittle. I’ve already replaced it, but it seems to break with minimal pressure. Guess it’s getting changed again. I find out my valve cover is made of plastic and there’s a screw with the head broken off in it that holds the PCV valve..

I remember why I hate Audi.

I start the car, take it for a drive, everything seems to be OK until I get about 10 miles out.. the car slowly starts overheating. I try to bleed the outrageous coolant system. It doesn’t help. I notice there’s no water flow unless you rev the engine.. with a bled system a sure sign of a bad water pump. Which means you get to take the entire nose of the car off so you can change the TIMING BELT DRIVEN WATER PUMP!

Yay.

Did I mention the OE water pump impeller is made of plastic and all the replacements are cast iron? Gee, I wonder why.

Thanks Audi. Clearly I was just jealous all these years. The cuts all over my hands are a testament to that. I’m so glad I finally own some kind of Audi.

In all seriousness though, it seemed to be very reasonably priced for a comfortable “AWD” (It’s not really AWD, don’t let that Quattro stuff fool you — the rear diff is open!) vehicle, and I couldn’t drive my daughter around in snowy weather in a rear wheel drive car with large tires — because they just act like big sleds and you don’t get to go anywhere.

This story continues with a horrible tale of a timing belt replacement.

And I ask myself — why did my 1979 Firebird need less maintenance and why was it literally over one hundred times faster to fix mechanical issues on it?

HP ProLiant DL580 G5 — GOOD LORD HOW LOUD YOU ARE, Also: How to quiet down a DL580 G5

So these servers I got are insanely loud, and I can’t stress insanely enough. The fans don’t seem to spin down to any reasonable level even when nothing is stressed and the system as drawing a “paltry” 650w.

So since there’s no direct control of the fans, at least so far in my limited testing with FreeBSD which I’m unfamiliar with anyhow I decided to quiet things down the hardware way. I took each of the 6 120mmx120mmx38mm fan cages out with *65dB* fans rated at 150CFM and cut the power cables to the proprietary connector. There was no way to get a standard width fan in here easily, so I decided to try running each pair in series.

That didn’t go so well. The fans would spin up a bit and then spin all the way down, the server thought they were bad. 6 volts was not enough to keep them going.. so I decided to cheat another way. I cut the power wires to some of the remaining 4 and wired a few diodes in series (5 of the 6 fans have 2 diodes, one has 3). That should give me some voltage drop from the forward bias of the diodes, and it did! My system went from absurdly loud to manageable.

For the other server I’ll be trying 4 diodes in series to try to get a bit more Vdrop and a bit more manageable noise. So far, so good. The 1.5amp diodes aren’t quite enough for my 150CFM Deltas and managed to burn out. 4x 3 amp barrel diodes fit perfectly in the little cavity in the fan, but I don’t have any pictures to show of that at the moment unfortunately.

 

 

3 1.5amp Rectifier Diodes, Twisted Together

3 1.5amp Rectifier Diodes, Twisted Together

Step 1:

Twist the diodes together, anode to cathode (note the silver band), this will get us some voltage drop.

3 1.5amp Rectifier Diodes, Soldered

3 1.5amp Rectifier Diodes, Soldered

Step 2:

Solder these connections and clip the parts we just soldered, we just need the two ends. This step is basically the same for 4 diodes, you just cut one more twisted set.

Fan and cage separated

Fan and cage separated

Step 3:

If your fan has a cage, disassemble the fan from the cage. Mine had plastic push pins much like most cars do, after that it slipped right out as I spread it apart to take the custom connector out.

Diodes Placed on Fan, Tinned

Diodes Placed on Fan, Tinned

Step 4:

Place the diodes on the fan, you can use some super glue to hold them in the cavity if there is a cavity. Otherwise place them somewhere convenient. Tin the anode/cathode of the diode(s) and cut the main power wire to the fan. Tin those wires.

Diodes Wired To Fan

Diodes Wired To Fan

Step 5:

Solder the power wires. The incoming wire will be wired to the cathode, or to the diode with the band furthest away from the connection.

Assembled Fan

Assembled Fan

Step 6:

You should probably have used shrink wrap in Step 5 (doh!), use some electric tape to make sure the exposed power does not hit ground and short out. (Yes, I had this problem, even with the electric tape, and I had to do horrible things to get the fan going again since it uses an uncommon connector that I couldn’t just replace)

Servers!

I got a pair of HP Proliant DL580 G5s, old, but potentially good. I was looking for something to run the ZFS SAN setup (which will hereafter be referred to as Hermes), and perhaps another VM server. They were supposed to have 2 Xeon E7330 quad-core CPUs and 32GB of DDR2 RAM.

To my surprise when I booted them up, server one had:

  • 4x X7460 CPUs (Six-core, 2.66GHz 16MB L3, SSE4)
  • 128GB of RAM

The second also had 128GB of RAM, but 4 E7450 CPUs, also six-cores but at a slightly lower 2.4GHz clock speed and I believe less cache. Total score, a single X7460 is worth more on eBay than I paid for both servers, so I ordered some cheaper CPUs (you know — the cheapest I could find that would work in the socket) at around $6/ea and these will go on eBay to help fund my little lab.

Once the MSA70 comes in I’ll be moving all the SAN stuff off the old desktop it’s in now and onto Hermes.

Note: These things SUCK power. Like 650w sitting there doing nothing power usage. But hey.. 128GB of RAM. That’s a lot of ARC for my ZFS machine!

How My Network Broke Today (Part I of atleast a billion)

So today I went to spin up a new VM for development use. It wouldn’t get an IP address, I saw the DHCP request on the DHCP server, and saw an offer go out but it was never received.  I dug through, and it seemed like this was just happening on one VLAN since everything else was OK.

Did I mention everything else was already running?

Did I mention if I had a trap collector with an alarm board that I would know what had happened almost immediately and been able to pinpoint the issue before I even saw the effects?

No? Well, now I have.

Let’s just say that I spent over an hour digging, running tcpdump on various interfaces, then finally hit the switches. I noticed there was only one port in the port channel on the Dell 5224 access switch when there should have been two down to the distribution switch. Odd but I thought inconsequential (at the time).

I got into the Cisco switch and saw MAC flaps (TRAPPABLE) all over the place with Po2. Odd again. The Dell switch must be to blame, so I go back to it and shut the port that’s not in the LACP port channel but should be. Things improve. Have I mentioned that I’d unplugged that fiber a week ago and only recently got a new one to plug back in?

I spend some time trying to get both ports in the port channel to no avail. I finally look at the config and notice the VLAN allowed config is slightly off (one is missing from eth 1/23), so I shut both the ports on the Cisco side as Dell won’t let you change interface configs while it’s part of a port channel and this was just faster — I reset the eth 1/23 config to match eth 1/24, and voila both ports came up.

But things were even worse now, barely any MACs were seen in ‘show mac address-table’ on my 3550-12 from Po2. And they were all on VLAN 1. Ugh. I shut the interfaces again and reset some more of the configuration on the Dell switch. I pray. (I don’t really pray). I bring the interfaces back up and all is good. The VM gets its IP address and everything is right in the world.

I really hate the Dell configurations. If I hated this switch before it’d be an understatement, and it’s only given me more of a reason to want to smash it with a hammer today. It’s mainly due to me not being familiar with them, but their configs aren’t as intuitive as I’d like.

iSCSI Booting Win2012 Server WITHOUT an HBA (Intel I350-T2 / 82571 / 82574 etc)

Thankfully Intel cards have iSCSI initiators in their firmware, so I setup a ZFS volume to make my HTPC diskless to attempt to stress the file server a bit more and generally just play with things as I tend to do.

So I added some settings to my ISC DHCP daemon under my shared network stanza to pass IQN/server settings to the Intel I350 card (82574 etc would work equally well here):

shared-network "VLAN-451" {
 default-lease-time 720000;
 option domain-name "p2.iscsi.frankd.lab";
 option domain-name-servers ns.frankd.lab;
  subnet 172.17.2.128 netmask 255.255.255.128 {
  range 172.17.2.144 172.17.2.239;
 }
 host intel-htpc1 {
  hardware ethernet a0:36:9f:03:99:7c;
  filename "";
  option root-path "iscsi:172.17.2.130::::iqn.2014-12.lab.frankd:htpc1";
 }
}

Voila, the card came up, grabbed DHCP settings and immediately initiated a connection! Awesome, the first thing to go right so far!  I admit I briefly spent some time trying to get iPXE to work with the Realtek card, but I ran into issues and just decided to use something I had laying around to get up and running quicker. The onboard Realtek is now for regular network data only, I might get a single port Intel card since I don’t need MPIO to this machine.

I imaged Win2012 Server to a USB stick using Rufus and plugged it in, it saw the drive and installed to it. I can’t believe things are going so easy/well for once! Then the system reboots. And it mounts the volume. And the Windows logo comes up. Then an error message comes up saying it couldn’t boot. Right away I knew it wasn’t getting past the BIOS calls to the disk (which were taken care of by the Intel NIC), and some Googling came up with horrible answers until I found an IBM document saying a new Intel driver fixes the issue — in a very indirect way. They don’t specify what, but it apparently has something to do with the iBFT tables that are created for the handoff. So I downloaded the newest drivers, put them on the USB stick and I installed Windows 2012 Server AGAIN. This time I loaded the newest version of the network drivers off the USB stick before even partitioning the disk, though.

The machine rebooted..

 

And..

 

IT WORKED! I was up and running. I installed the User Experience stuff so I could get Netflix/Hulu up easy, downloaded nVidia drivers and am now getting my Steam games downloaded to the machine — although I could stream off my workstation/gaming PC. It can’t hurt to have more than one machine with them installed in case either one of them dies and I need to go blow some pixels up to relieve some stress though, right?

 

Getting My Real VM Server Back Online Part III: Storage, iSCSI, and Live Migrations

After some dubious network configurations (that I should have never configured incorrectly) I finally got multipath working to the main storage server. All of the multipath.conf examples I saw resulted in non-functional iSCSI MPIO, while having no multipath.conf left me with failover MPIO instead of interleaved/round-robin.

A large issue with trying to get MPIO configured was the fact that all the examples I found were either old (and scsi_id works slightly differently in Ubuntu 14.04) or just poor. Yes, I wound up using Ubuntu. Usually I use Slackware for EVERYTHING, but lately I’ve been trying to branch out. Most of the VMs run Fedora, “Pegasus” or VMSrv1 uses Fedora, “Titan” uses Ubuntu.

Before I did anything with multipath.conf (It’s empty on Ubuntu 14.04), I got this:

root@titan:/home/frankd# multipath -ll
1FREEBSD HTPC1-D1 dm-2 FREEBSD,CTLDISK
size=256G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 13:0:0:0 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 12:0:0:0 sdd 8:48 active ready running

Note the disks are both round-robin — with only one member each! This works for fail-over, but did nothing for performance. The only thing that wound up working for multipath.conf was this:

defaults {
 user_friendly_names yes
 polling_interval 3
 path_grouping_policy multibus
 path_checker readsector0
 path_selector "round-robin 0"
 features "0"
 no_path_retry 1
 rr_min_io 100
}

multipaths {
 multipath {
  wwid 1FREEBSD_HTPC1-D1
  alias testLun
 }
}

The wwid/alias doesn’t work, however. All of the MPIO is just coming from the defaults stanza. I attempted many things with no luck, unfortunately. I’m going to have to delve into this more especially if I want live migrations to work properly with MPIO. As it stands the disk devices are pointing at a single IP (ex /dev/disk/by-path/ip-172.17.2.2:3260-iscsi-iqn.2014-12.lab.frankd:htpc1-lun-0), I’ll need to point at aliases to get the VMs working with multipath.

The multipath tests themselves were promising though, dd was able to give me a whopping 230MB/s to the mapper device over a pair of GigE connections.

The output from ‘multipath -ll’ now looked more reasonable:

root@titan:/home/frankd# multipath -ll
mpath1 (1FREEBSD HTPC1-D1) dm-2 FREEBSD,CTLDISK
size=256G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 39:0:0:0 sde 8:64 active ready running
  `- 40:0:0:0 sdg 8:96 active ready running

You can see the drives are both under the same round-robin policy instead of two separate ones.

The storage server also saw some slight changes, including upgrading from one Intel X25-V 40GB for L2ARC to 2xX25-Vs for a total of 80GB. I also added a 60GB Vertex 2 as a SZIL device. I really need to build a machine with more RAM and partition out the SZIL. I’ll likely wind up using my 840Pro 256GB for L2ARC and leave the old X25Vs out of the main array once I get a pair of 10GbE cards for maximum speed (hopefully near-native of the 840Pro — perhaps better with a large amount of ARC) to my workstation.

So we’re at a point where everything appears to be working, although in need of some upgrades! Great! I’m looking at a KCMA-D8 Dual Opteron C32 motherboard as I have a pair of Opteron 4184s (6 core Lisbon, very similar to a Phenom II X6 1055T) laying around, so I could put together a 32GB 12 core machine for under $400 — but as always, budgetary constraints for a hobby squash that idea quickly.

Getting My Real VM Server Back Online Part II: Storage Server!

Anticipating the arrival of RAM for my VM server tomorrow I decided I needed some kind of real storage server, so I started working on one. I haven’t touched BSD since I was a kid, so I’m not used to it in general. I wasn’t sure how OpenSolaris would work on my hardware (I hear it’s better on Intel than AMD) so I opted for FreeBSD. Unfortunately I just found out FreeBSD doesn’t have direct iSCSI integration with ZFS, but that’s okay! We can always change OS’s later, especially since the storage array leaves a lot to be desired (RAID-Z1 with 4x1GB 2.5″ 5200RPM drives + 40GB Intel X25V for L2ARC, no separate ZIL).

I’m getting used to the new OS and about to configure iSCSI, which will be handed out via multipath over an Intel 82571EB NIC into two separate VLANs into a dedicated 3550-12T switch. We’ll see how it works, and if it’s fine I’m going to get my HTPC booting over it.

I’m going to look around for a motherboard with more RAM slots, for now I’m stuck with a mATX motherboard, a SAS card that won’t let the system boot, and 2 RAM slots (8GB) with an FX-8320.

Performance tests to come.. after I encounter a dozen issues and hopefully deal with them!

Rearranging The Intranet of Things Part II

I’m sure there will be a lot more posts like this to come. I had formerly moved the edge router to the ‘closet’ (aka the garage, right next to the cable modem and 3560-24PS sitting there) and added another router there to have a routed gig port into my ‘office’ (aka my bedroom with a couple desks).

Today I replaced both routers with a single 7206VXR with an NPE-G1. I had it all configured and everything should’ve worked off the bat, but it didn’t — not exactly, anyway. The routing was perfect, the NAT was great. But I only have a VAM card which doesn’t work with 15.x (only VAM2 cards work with new code), and I didn’t want it doing VPN in software.

So I decided to keep the old WAN router as VPN-only duty. I briefly considered using a 1760 with a VPN module (I have a few), but when I finally get to having decent internet speeds it would choke. The 3825 has an EPII+ card on top of the onboard hardware engine, so it should at the least have no issue keeping up with my internet connection with weak Triple-DES. The only issue is when I went to forward UDP 4500 from the edge router to the VPN router I got:

% Port 4500 is being used by system

I was able to successfully forward ports UDP 500 and ESP, but here I got stumped. I verified there was no crypto config, I tried clearing crypto stuff, I tried disabling software crypto — all with no luck. Googling didn’t give me much to go on, but I finally ran into something showing this error as an IOS-XE bug for 15.2(4)S2 –and I was running 15.2(4)S3 (pure IOS, but basically the same), so being out of options and ideas I decided to just install 15.2(4)M7 and Voila! Problem solved!

Two routers replaced with — two routers, maybe that doesn’t sound very good, but it will allow me to do more at the edge with more ports available directly on the router instead of playing with switches and VLANs/VRFs.

And in case you want to see how my network is physically wired — and this is somewhat simplified, here you are!

Network Diagram

Simplified Network Diagram – 01/01/15

Rearranging The Intranet of Things

So after dealing with a bunch of random dd-wrt based access points I decided to grab some LAP1142Ns off of eBay. I set up a vWLC on the VM machine, and was able to get it going fairly quickly even with no knowledge of Cisco Wireless technology.

So far my throughput is only slightly increased even after moving to 5GHz and having a 3×3 MIMO radio in my laptop.

I added a real router for the upstairs network (3825), and a gig link from the ‘closet’ to my office/workstations. Some of the interconnects in the lab are temporarily dual 100MBit load balanced via EIGRP to alleviate some of the bottlenecks. The LAP1142Ns are limited to 100mbit due to a 3560-24PS being the only POE switch I have, but I never see more than about 60mbit of throughput over wireless, and the port never exceeds 70mbit — so until I get that sorted out it’s not a limitation.

To get more gig links in my ‘office’ (aka my bedroom) I trunked a cheap Dell 5224 to a 3550-12G, replacing the 3550-12T that was formerly there. I wish I could afford newer Cisco gig switches my budget is basically non-existent.

I still need a total network redesign, my routing table is almost laughable:

dswr1.core#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2
 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
 ia - IS-IS inter area, * - candidate default, U - per-user static route
 o - ODR, P - periodic downloaded static route

Gateway of last resort is 172.16.5.6 to network 0.0.0.0

D 192.168.30.0/24 [90/28928] via 10.255.1.6, 22:58:10, FastEthernet0/16
 [90/28928] via 10.255.1.2, 22:58:10, FastEthernet0/14
 172.17.0.0/16 is variably subnetted, 6 subnets, 2 masks
D 172.17.0.48/28 [90/28672] via 10.255.1.6, 1d00h, FastEthernet0/16
 [90/28672] via 10.255.1.2, 1d00h, FastEthernet0/14
D 172.17.0.32/28 [90/28672] via 10.255.1.6, 1d00h, FastEthernet0/16
 [90/28672] via 10.255.1.2, 1d00h, FastEthernet0/14
D 172.17.0.16/28 [90/28672] via 10.255.1.6, 1d00h, FastEthernet0/16
 [90/28672] via 10.255.1.2, 1d00h, FastEthernet0/14
D 172.17.0.0/28 [90/28672] via 10.255.1.6, 1d00h, FastEthernet0/16
 [90/28672] via 10.255.1.2, 1d00h, FastEthernet0/14
D 172.17.0.72/29 [90/28672] via 10.255.1.6, 1d00h, FastEthernet0/16
 [90/28672] via 10.255.1.2, 1d00h, FastEthernet0/14
D 172.17.0.64/29 [90/28672] via 10.255.1.6, 1d00h, FastEthernet0/16
 [90/28672] via 10.255.1.2, 1d00h, FastEthernet0/14
 172.16.0.0/16 is variably subnetted, 7 subnets, 4 masks
C 172.16.255.0/28 is directly connected, Vlan601
D 172.16.2.8/30 [90/28416] via 10.255.1.6, 1d00h, FastEthernet0/16
 [90/28416] via 10.255.1.2, 1d00h, FastEthernet0/14
D 172.16.2.4/30 [90/28672] via 10.255.1.6, 22:58:18, FastEthernet0/16
 [90/28672] via 10.255.1.2, 22:58:18, FastEthernet0/14
C 172.16.5.4/30 is directly connected, FastEthernet0/24
D 172.16.3.2/32 [90/156672] via 10.255.1.6, 22:58:14, FastEthernet0/16
 [90/156672] via 10.255.1.2, 22:58:14, FastEthernet0/14
D 172.16.1.0/24 [90/28672] via 10.255.1.6, 1d00h, FastEthernet0/16
 [90/28672] via 10.255.1.2, 1d00h, FastEthernet0/14
D 172.16.3.1/32 [90/156160] via 172.16.5.6, 10:49:53, FastEthernet0/24
 172.18.0.0/28 is subnetted, 1 subnets
D 172.18.0.0 [90/28672] via 10.255.1.6, 1d00h, FastEthernet0/16
 [90/28672] via 10.255.1.2, 1d00h, FastEthernet0/14
D 192.168.99.0/24 [90/28928] via 10.255.1.6, 03:03:00, FastEthernet0/16
 [90/28928] via 10.255.1.2, 03:03:00, FastEthernet0/14
 10.0.0.0/30 is subnetted, 2 subnets
C 10.255.1.4 is directly connected, FastEthernet0/16
C 10.255.1.0 is directly connected, FastEthernet0/14
D 192.168.0.0/24 [90/30720] via 172.16.5.6, 10:49:54, FastEthernet0/24
D 192.168.100.0/24 [90/28672] via 10.255.1.6, 1d00h, FastEthernet0/16
 [90/28672] via 10.255.1.2, 1d00h, FastEthernet0/14
C 192.168.101.0/24 is directly connected, Vlan400
D*EX 0.0.0.0/0 [170/30720] via 172.16.5.6, 10:49:54, FastEthernet0/24

Getting My Real VM Server Back Online

My server has been off hiding somewhere far away from me for a while, so I’ve been running virtual machines on an AMD FX-8320 990FX based box. Unfortunately it only had 16GB of RAM and I gutted the server RAM for use in my workstations.

I’ve decided to order some used ECC Registered 4GB sticks off of eBay — 32GB ought to do for now. I won’t have to worry about whether I can launch a new VM due to RAM constraints (I was using a lot of swap before!), so titan.frankd.lab will soon be back online with the FX-8320 machine for failover. I’m going to need shared storage, so I’ll have to setup a real iSCSI storage box soon.

End short random thought.