The 10kΩ That Wasn't
I ordered 200 Vishay Z-foil 10kΩ 0.01% resistors for a precision voltage reference prototype. Standard stuff—I'd read the datasheets, checked the TCR specs, everything looked solid. When they arrived, I grabbed my trusty Fluke 117 multimeter and started sampling.
The first resistor read 10.04kΩ. Second: 10.01kΩ. Third: 10.07kΩ. By the tenth, I was already dialing Vishay's support number. How could 0.01% parts show 0.4–0.7% deviation? Did they send the wrong tolerance? Was their quality control slipping?
I didn't sleep well that night. The project deadline was two weeks away, and I was staring at a box of components I no longer trusted.
What I Missed: The Measurement Trap
Everything I'd read about precision resistors said they're only as good as your test gear. I thought I understood that. But I didn't really get it until I wasted $3,200 and a month of project time chasing a phantom problem.
1. The Multimeter's Resolution Floor
The Fluke 117 is a fine meter for general use—I still carry one in my bag. But on the 40kΩ range, its best resolution is 10Ω. A 0.01% resistor of 10kΩ has an absolute tolerance of ±1Ω. So the meter can't even see the difference between a perfect part and one at the edge of spec. The 0.04–0.07kΩ variation I saw was just digit noise and range switching artifacts.
2. Two-Wire vs. Four-Wire: The Elephant in the Lab
I was using standard test leads with alligator clips—two-wire measurement. On a 10kΩ resistor, the lead resistance (roughly 0.15Ω per lead) contributes 0.003% error. Negligible, right? Except the problem wasn't the wire resistance; it was the contact resistance. A slightly corroded clip, a tiny oxide layer on the resistor lead, and suddenly you've got 50mΩ of variable contact resistance that changes every time you touch the part. For a 1Ω resistor, that's 5% error. For a 10kΩ part, it's only 0.0005%—but the real issue is repeatability. Every reading was different because the contact was inconsistent.
3. Self-Heating and Thermal EMF
I also ignored the fact that the Fluke 117 sources about 100µA for resistance measurement on that range. That 100µA through 10kΩ dissipates 0.1mW—enough to raise the resistor's internal temperature by a fraction of a degree. With Vishay's Z-foil TCR of ±0.2ppm/°C, that's a shift of maybe 0.002Ω. But thermal EMF from the lead connections (especially when I held the clips with my fingers) could inject microvolt-level offsets that the meter interprets as extra ohms. Another subtle, invisible error.
(Oh, and I should add: I was measuring on a metal workbench that had temperature gradients. The more I think about it, the more I realize how many small mistakes added up.)
The Real Cost: Not Just Money
I returned the entire batch—200 resistors—and ordered replacements from a different brand. The return shipping, restocking fee, and rush freight for the new order totaled $890. But the bigger hit was the schedule: a three-week delay while I re-qualified the new parts, redesigned the PCB footprint slightly, and re-ran simulations. The project sponsor was not happy. I'd estimate the total cost, including my engineering time, at around $3,200. All because I trusted a hand-held meter to verify a precision component the wrong way.
And of course, the new brand's resistors showed exactly the same variation on the Fluke 117. The problem was never the resistors.
How to Actually Verify Vishay Precision Resistors
I've now implemented a simple pre-check procedure for any precision component order:
- Use four-wire (Kelvin) measurement for any resistor with tolerance ≤0.1%. A typical 6.5-digit DMM like the Keysight 34465A or Keithley 2010 can resolve 0.001Ω, and the four-wire connection eliminates lead and contact resistance.
- Wait for thermal equilibrium. Let the resistor sit on the test fixture for 30 seconds before reading. Even better, use a low-power measurement (e.g., 10µA excitation) to minimize self-heating.
- Check the temperature of your test environment. Vishay’s foil resistors have superb TCR, but the measurement itself must be stable. A 1°C drift in ambient can affect some meters' reference voltage.
- Cross-check with the manufacturer's test report. Vishay includes a data sheet with each precision resistor lot showing actual measured values. Trust that data unless you have a calibrated, traceable setup.
According to Vishay's application note "Measuring Precision Resistors" (AN-302), the recommended method for verifying 0.01% tolerance is to use a 6.5-digit multimeter with four-wire sensing and a low-thermal-emf fixture. Anything less will give you noise, not data.
The Honest Limitation
If your application only needs 1% resistors, a Fluke 117 is completely sufficient—you'll never notice the measurement imperfections. But if you're pushing down to 0.1% or below, you need to invest in the right test setup, or trust Vishay's own data. There's no shame in saying a product is overkill for your current measurement capability.
I recommend Vishay precision foil resistors for designs where temperature stability and long-term drift matter—like precision voltage references, metrology, and medical instrumentation. If you're building a consumer gadget with 5% resistors, these parts are wasted. And if you can't measure them properly, you'll mistake their excellence for a defect.
That's a lesson I learned the hard way—and a checklist I now keep on my lab wall.