is it possible these will work if I just copy them back over after the upgrade?

No! You will need to tweak them a bit. Basically copy your b2evo files and the database, upgrade that copy, fix all errors, make sure everything is working, and only then make it live.
It will take some time, trust me.

I'm not sure I understand. My plan was to backup those files, then upgrade by deleting everything on the server (DB NOT included) and uploading the new version, run the install script, then copy my backed up files back to the server. Is this the right route? And if so, what kinds of errors can I expect to see? I'm no PHP coder, will I be in for a nightmare? I'm tempted to just leave it alone since I've got a ton invested in this site but I hate to do that... And how to keep it NOT live? If I go into maintenance mode I won't be able to tell what kinds of errors I'm dealing with right?

Do what you wrote, but not on your live website. Make a copy of your entire b2evo directory (website) in a subfolder or subdomain, also copy the database. I mean create a second copy of your b2evo, and work on it before you get everything straightened up.

You will see some errors in your older skin(s). Do not copy old files from /skins directory, only backup the actual /skins/yourskin folder. Some plugins may not work in v4.1

I recommend you to work on a duplicate instance so you can still use your original v3.x website while fixing and upgrading b2evo.

It's not a huge lot of work to upgrade from v3 to v4.1, it's just easier that way ;)

http://manual.b2evolution.net/Upgrade_from_an_older_version#General_procedure_for_upgrading_from_a_previous_version_of_b2evolution

Ok, that makes perfect sense - thanks! I'll get started and hope for the best!

Well, everything went great! Was able to port over 2 skins, a bunch of plugins and some jquery stuff with no problems! Thanks for your help! Mission Success!

:)