Matching neutral fabrics to your neutral color swatches can take a lot longer than matching reds, blues or greens.
Here's the explanation, I believe: We think in language. But we lack an extensive vocabulary to talk about neutrals, so it's literally harder for us to think about what we're seeing when we look at a neutral. (Here's a cool Wikipedia page with more info about thinking about color.)
For example, if you want to match a blue, you can tell yourself to look for a cadet blue or a Chinese blue or a sky blue. Each of those terms is perhaps attached in your mind to a specific kind of blue.
But if you want to match a white, it's harder to articulate to yourself the kind of white you're looking for. We just don't have a ton of words to describe shades of white.
What do we call white with green, or white with yellow, or white with pink?
I mean, I have answers to some of those questions, but only because I've been thinking about color for a while now. The average person doesn't learn all of that vocabulary as part of normal life.
So when you're looking for a matching white, you may not be able to talk to yourself about the kind of white you're looking for. And if you don't have language for a color, it's almost impossible to think about it.
To get around this, I recommend asking yourself a couple of questions:
1. What basic hue seems to tint this neutral?
2.a. (for whites) Does it look more dirty and sooty, or more pure and clean?
2.b. (for other neutrals) Is it more light or more dark?
(If it's not obvious, these questions are a way of getting at the hue, value, and saturation of a neutral.)
If you ask and answer these questions about a particular white, you might call it "a dirty white with a bit of blue in it."
Similarly, a grey can be light or dark, yellowish or blueish; a brown can have pink or yellow in it; etc.
If you can identify a few adjectives to attach to your neutral -- in other words, if you can say, "a sooty, pinkish white" or "a light brown with some purple in it" -- you're much closer to being able to match that neutral. Say the words to yourself in your head. Having actual language for the color really helps!
If you don't own your seasonal palette, color cards are an affordable way to get your hands on it!