That’s more of a bug instead of someone actively monitoring you. The device accidentally thought you activated it, so it started listening.

You wouldn’t be able to access those recordings if they were trying to spy on you.

Besides that, you literally agreed to it when buying and setting the device up. This is not the case with your phone (if you switch the assistant off, if it’s on and heard the keyword it might still upload data of course).