Presuming you mean "Hotel Union Square" (subtle difference in names, but I'm 99% sure that's what you're talking about), Google maps shows the walk as being a hair over 1 mile (1.1 to be precise) and estimate it at ~20 minutes. It's a very straightforward, level, walk so it's pretty much the time, distance & luggage to consider (rather than say a major elevation change). There's a little details on your walking option.
Public transit is the other option which means either BART (as simba pointed out) or MUNI. Either one will run from the Powell to Embacardero, so the walking there is identical (well, actually the BART tracks run underneath the MUNI tracks, so BART would have a little more elevation change, but it's really nothing to speak of).
Sunday means you don't have any rush hour considerations. If you're getting a MUNI Passport, I'd use it (if you still have a day on it) and take MUNI. If not, you might check the BART schedule and see when trains are due (BART runs very much on schedule) and how it fits in with your timing (and then see how it looks when you're actually ready to go). MUNI doesn't have a schedule as much as a "frequency" and, depending on the BART schedule might mean a shorter average wait. Don't have the MUNI versus BART fares in front of me, but I can't imagine the difference would be anything significant.
With the transit options you're probably taking a few minutes walking at both ends and maybe a 5-10 minute & ride. I'd say 10-15 minutes total if you take public transit.
Lastly and I don't know all the details, but I'm pretty sure that Amtrak buses make stops at a handful of hotels in the downtown area (in addition to the Ferry Building). You might check this out, it's possible you could catch the bus at one of these hotels and just have a couple of block walk rather than having to go all the way to the Ferry Building. (Which you might or might not want to do.)
Hope this helps,