You have either explicitly (with FLOCK( )) or implicitly (with REPLACE ALL, for example) attempted to take a file lock in a transaction. This is illegal if any record locks have already been taken either explicitly or implicitly at this transaction level or any lower transaction level. This is illegal even if those record locks have been freed inside the transaction.
You must roll back all record locks on this file before attempting this file lock. For more information, see ROLLBACK.