Colorful clothes gives me anxiety and panic attacks, iin?

Is it weird that this makes me want a blue ring octopus outfit?

Makes total sense and now I've got some Bell Biv DeVoe in my head.

Do you have the brain of a rattlesnake?

ʇuǝɯǝɔɹoɟuǝ ʍɐl ɟo pıɐɹɟɐ ǝɹɐ noʎ ʞuıɥʇ ı ˙ɯıɥ ʇooɥs oʇ pǝıɹʇ ʇuǝƃɐ uɐ uǝɥʇ ˙ʞɔɐlq uıɐld ǝɹoʍ ɹǝɥ punoɹɐ ǝldoǝd ǝɥʇ llɐ puɐ ssǝɹp pǝɹ ʇɥƃıɹq ɐ ǝɹoʍ ʎpɐl ǝɥʇ ǝıʌoɯ xıɹʇɐɯ ǝɥʇ uı

Your reason of bright and colorful creatures being poisonous is correct and nothing in the word is irrational.
As you know that humans are non toxic and so are their clothes so I would suggest limiting your fears of poison to animals.
If it's difficult to do this alone see a psychologist. It will provide you an extra support to explain it to yourself
Wish you the best. :-)

Might be you're just super self-conscious about wearing bright clothes yourself, and you push that fear towards others. We project our feelings and fears onto other people all the time.

Look, in your case I would avoid black people during warm weather.