Hello OutSystems Community,
I’m reaching out for help regarding a persistent issue I’ve been facing with converting an Excel file generated by an Azure Function into a Record List in OutSystems. Despite numerous attempts to resolve the issue, I keep encountering the error that the result cannot be shown in the frontend and the debugger becomes unresponsive by ExcelToRecordList.
Background:
I’ve been working on an Azure Function that processes data and generates an Excel file. Below is a simplified version of the relevant code snippet:
This Azure Function creates an Excel file that I need to convert into a Record List so that it can be presented in the frontend. However, every time I try to perform this conversion using OutSystems, the system becomes responsive. By opening the file, a warning occurs: "this file is being verified by microsoft defender advanced threat protection" and the file is always opened in Protected View.
Steps Taken to Resolve the Issue:
Initial Validation:
Writing a cleanExcel function with different scripts and approaches in an extention in Integration Studio
Advanced Excel Component:
Manual Inspection and Editing:
Testing with Entity Bootstrapping:
Current Situation:
After exhausting all known methods to rectify this issue, I am at a standstill. Despite the file appearing to be a valid Excel file, OutSystems continues to reject it with no reaction, neither in the frontend nor by debugging.
Request for Assistance:
I’m open to any advice or insights you may have. I’ve attached the problematic Excel file for reference.
Unfortunately I didn't make it after the whole week and the hard deadline before the deployment is the 6th September.
I sincerely appreciate for your help!
Best regards,
Have followed another approach - copy all the sheets to new excel using the Nuget package "NPOI" . Not very efficient approach. Attached the code for your reference. Its works for me.
Could you please revise your Azure function code? It seems the issue is in the Python code used to generate the Excel file. Could you try using different libraries for file generation?
Also, when comparing the original file to the converted one, I noticed a difference: the original file has a Byte Order Mark (BOM), but the converted one does not.
Thank you for replying, Siya.
1. you mean like xlsxwriter instead of openpyxl? The python code was delivered by another team. It's hard for me to try with different approaches and debug. : /
2. So I could also try the BOM removal on the entire Excel file or each XML file by another extension?
Please see if you can remove the BOM on the entire excel.
I tried with the following script in different versions, somehow the BOM elements (UTF-16 LE BOM (FF FE) at position 26478 within sheet.xml and UTF-16 BE BOM (FE FF) at position 35684 within the same sheet.xml file. Both are embedded in the XML content, not in Binary Data) could not be removed. Do you have a clue?
public void MssremoveBomFromExcel(byte[] ssinputExcel, out byte[] ssoutputExcel)
{
try
using (MemoryStream outputMemoryStream = new MemoryStream())
using (MemoryStream inputMemoryStream = new MemoryStream(ssinputExcel))
using (ZipArchive inputArchive = new ZipArchive(inputMemoryStream, ZipArchiveMode.Read, true))
using (ZipArchive outputArchive = new ZipArchive(outputMemoryStream, ZipArchiveMode.Create, true))
foreach (var entry in inputArchive.Entries)
var outputEntry = outputArchive.CreateEntry(entry.FullName);
using (var inputEntryStream = entry.Open())
using (var outputEntryStream = outputEntry.Open())
byte[] entryBytes = ReadAllBytes(inputEntryStream);
byte[] cleanedBytes;
if (IsTextFile(entry.FullName))
string encoding;
cleanedBytes = RemoveBomAndReencode(entryBytes, out encoding);
Console.WriteLine($"Processed {entry.FullName} with encoding {encoding}");
}
else
cleanedBytes = entryBytes;
outputEntryStream.Write(cleanedBytes, 0, cleanedBytes.Length);
ssoutputExcel = outputMemoryStream.ToArray();
catch (Exception ex)
throw new Exception("Failed to remove BOM from Excel content.", ex);
// Method to remove BOM from a byte array and re-encode if necessary
private static byte[] RemoveBomAndReencode(byte[] bytes, out string encoding)
encoding = "UTF-8"; // Default encoding
// Detect and handle UTF-16 LE BOM
if (bytes.Length >= 2 && bytes[0] == 0xFF && bytes[1] == 0xFE)
encoding = "UTF-16LE";
string content = Encoding.Unicode.GetString(bytes.Skip(2).ToArray());
return Encoding.UTF8.GetBytes(content);
// Detect and handle UTF-16 BE BOM
else if (bytes.Length >= 2 && bytes[0] == 0xFE && bytes[1] == 0xFF)
encoding = "UTF-16BE";
string content = Encoding.BigEndianUnicode.GetString(bytes.Skip(2).ToArray());
// Detect and handle UTF-8 BOM
else if (bytes.Length >= 3 && bytes[0] == 0xEF && bytes[1] == 0xBB && bytes[2] == 0xBF)
encoding = "UTF-8";
return bytes.Skip(3).ToArray(); // Already in UTF-8
// If no BOM detected, return the original byte array
return bytes;
// Method to determine if a file is text-based
private static bool IsTextFile(string entryName)
return entryName.EndsWith(".xml", StringComparison.OrdinalIgnoreCase) ||
entryName.EndsWith(".rels", StringComparison.OrdinalIgnoreCase) ||
entryName.EndsWith(".txt", StringComparison.OrdinalIgnoreCase);
// Method to read all bytes from a stream
private static byte[] ReadAllBytes(Stream stream)
using (MemoryStream ms = new MemoryStream())
stream.CopyTo(ms);
return ms.ToArray();
// MssremoveBomFromExcel
Indeed! It also worked by my side, the excel file is no longer opened with Protected View and can be directly bootstraped into OutSystems. Many thanks, Siya!