This looks like it will be the one of the least impactful weakness in the entire game. Let's assume a NotZ chaos bag on Standard for The Gathering (no Blurse tokens). Assuming there is one Ghoul at your location, the probabilities of passing a test at 0 / 1 / 2 / 3 / 4 above difficulty are 25 % / 63 % / 81 % / 88 % / 94 % respectively without the Elder Sign, this changes to 20 % / 60 % / 80 % / 87 % / 93 % (probabilities calculated with www.arkhambagcalculator.net.) The change is -5 % / -5 % / -1 % / -1 % / -1 %. This is almost negligible.
Let's take a harsher bag into account (Devourer Below on Hard), the probabilities are 22 % / 33 % / 44 % / 67 % / 78 % and change to 18 % / 29 % / 41 % / 65 % / 76 %, a change of -4 % / -4 % / -3 % / -2 % / -2 % so even here, the impact isn't enormous.
So if you take 20 skill tests in the time it is active, you should at worst expect to lose one of your skill tests due to this card on which you would have succeeded otherwise. I guess it might be annoying when you draw it early on an Agenda with a high doom threshold (for example in Where Doom Awaits).