With only four days I would say a definite no. You could take the Victoria Clipper from Seattle. Not sure what your interests are, but even a visit to the Northern Section of Olympic National Park IMO would be more rewarding. In either case an overnight would make more sense.
Time of year makes a big difference as the days start getting longer.
If your objective is a ferry ride, I'm wondering why specifically the Port Angeles-Victoria run is necessary. The standard ferry experience in Seattle is to take the Seattle-Bainbridge Island run from downtown across Puget Sound. The ride takes 30 minutes each way, runs every hour and offers a scenic voyage to a nice bedroom community with my favorite ice cream parlor (Mora Ice Creamery) on the other side. Restaurants, art galleries, distilleries, breweries, gardens, a casino and some history, too. You can ride as a foot passenger for about $7 or take the car for $30 round trip. If you wish you can take the one-hour trip to the Navy town of Bremerton for the same price. See www.wsdot.wa.gov/ferries/ for other options.
Versus driving 3 hours to Port Angeles (by way of the above Seattle-Bainbridge ferry) paying $75 (each way w/ reservation) for a 90 minute trip which runs only twice a day., Then drive 3 hours back to Seattle.
If the visit to Victoria is your goal, you can also take the 2+ hour Victoria Clipper from downtown Seattle for about $140 RT for a one day or overnight trip. You may have more of a chance to see orcas on this route, but no guarantees.
Just seems the trip via Port Angeles is going to take a lot more time away from your 4 days than you may expect. Good luck!
Thank you for your reply. Going in early May.
Do you have a passport?
Yup - one of those documents. I just think that sometimes people are tossing a trip together and forget about that part. I did that last summer - getting all organized to head to ONP and suddenly realized my passport was in the shop (getting renewed). It did actually come back a few days before our trip in the end, but we had reorganized our plans to stay in Canada by then.
Okay, here's an idea. Kenmore Air flies commercial regularly scheduled seaplanes from Lake Union (almost downtown Seattle) to the Victoria waterfront and tons more nearby destinations around Puget Sound. I think there are usually 3 flights a day back and forth to Victoria. About $150 each way. You'd probably be spending half that if you drove but maybe that would be worth it if you really want to see Canada. Plus you'll get a great view of the area from a seaplane.
Definitely bring your passport or at least check out Canadian and US State Department websites to make sure you can get in and back. Canada has been turning back anyone with even a DUI conviction. Have a great trip.