Hi,
If it is your local list variable which holds your uploaded files. Then yes we can delete it. We don't need its ID. instead, you can pass the file position as an input parameter to the action while clicking that link.
Inside the action, you can use the above list remove method as João Marques mentioned. and pass the position to this action from the input parameter.