No. They still track you even if you don't tap into the filial's magic. I think they have a sense for anyone who has contacted the filial spirits. Or-or who know how to summon things, or are affected by a shift in general.