I don't know how to change it back.

Safari > preferences > safari opens with > a new window

Are you talking about the menu bar that starts with the Apple Logo? I have a mostly pink desktop background and consequently my menu bar background is pink with black letters.

My safari window toolbar is white with black letters.

Dark mode makes my menu bar kind of black with grayish lettering.

When I leave that site it goes back to normal.

It's new to Safari ...

https://css-tricks.com/safari-15-new-ui-theme-colors-and-a-css-tricks-cameo/

But old in Android, so many sites are setting it already:

https://developers.google.com/web/updates/2014/11/Support-for-theme-color-in-Chrome-39-for-Android

To change it you'd probably have to use a custom CSS that overrides "theme-color". I'd guess
that Safari will add a preference setting to do this in some future update.