50
Views
6
Comments
Solved
Issues Converting Excel into Record List: Debugger Became Unresponsive
Application Type
Reactive, Service
Service Studio Version
11.54.71 (Build 63499)

Hello OutSystems Community,

I’m reaching out for help regarding a persistent issue I’ve been facing with converting an Excel file generated by an Azure Function into a Record List in OutSystems. Despite numerous attempts to resolve the issue, I keep encountering the error that the result cannot be shown in the frontend and the debugger becomes unresponsive by ExcelToRecordList.

Background:

I’ve been working on an Azure Function that processes data and generates an Excel file. Below is a simplified version of the relevant code snippet:

  • pythonCopy code@app.route(route="download")async def download(req: func.HttpRequest) -> func.HttpResponse:    try:        files, user_id = await extract_files_and_user(req)        file_info_list = await prepare_file_info(files)        excel_files = [file for file in file_info_list if file['name'].endswith('.xlsx')]        excel_file = excel_files[0]                if not excel_files:            return func.HttpResponse(f"No Valid excel file", status_code=400)                result_excel = await process_results(excel_file, user_id)        if not result_excel:            return func.HttpResponse(f"Failed to process results for {excel_file['name']}", status_code=400)        output = io.BytesIO()        result_excel.save(output)        output.seek(0)        return func.HttpResponse(            body=output.read(),            status_code=200,            mimetype="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",            headers={                "Content-Disposition": 'attachment; filename="processed_checklist.xlsx"'            }        )      except Exception as e:        logging.error(f"An error occurred: {str(e)}")        return func.HttpResponse(f"An internal error occurred: {str(e)}", status_code=500)

This Azure Function creates an Excel file that I need to convert into a Record List so that it can be presented in the frontend. However, every time I try to perform this conversion using OutSystems, the system becomes responsive. By opening the file, a warning occurs: "this file is being verified by microsoft defender advanced threat protection" and the file is always opened in Protected View. 

Steps Taken to Resolve the Issue:

  1. Initial Validation:

    • I validated the file using the "Open XML Package Editor for Modern Visual Studios" and found nothing wrong with the file structure or content. Everything seemed correct, and no hidden errors or unexpected data were found.
  2. Writing a cleanExcel function with different scripts and approaches in an extention in Integration Studio

  3. Advanced Excel Component:

    • I attempted to use the Advanced Excel component from the Forge to clean or reprocess the file. This included:
      • Copying the worksheet to the same workbook and deleting the old one.
      • Using Excel_ClearProperties to clean all the unnecessary properties.
    • Unfortunately, none of these approaches solved the problem. The error persisted.
  4. Manual Inspection and Editing:

    • I manually inspected the XML parts of the Excel file to ensure there were no anomalies in the content types or the overall structure.
    • Even after making sure everything conformed to the expected standards, the issue remained unresolved.
  5. Testing with Entity Bootstrapping:

    • As a troubleshooting step, I tried bootstrapping the Excel file into an entity, but the error persisted, suggesting the file was not recognized as valid Excel.
    • I manually re-saved the excel file, the "new" file could then be bootstraped, which indicates all the structure labels matched the excel table headers
    • I manually opened the .xlsx file, duplicated the target worksheet, deleted the old one and saved the file.  The "new" file could also be bootstraped. 
  6. Trying different actions after ExcelToRecordList:
    • Assignment (Server Action Output Structure_RecordList = ExcelToRecordList) (ExcelToRecordList used the same structure. Sheet Name was left empty or filled)
    • ListAppendAll
    • ListAppend in a loop


Current Situation:

After exhausting all known methods to rectify this issue, I am at a standstill. Despite the file appearing to be a valid Excel file, OutSystems continues to reject it with no reaction, neither in the frontend nor by debugging.

Request for Assistance:

  • Has anyone encountered a similar issue when converting an externally generated Excel file into a Record List in OutSystems?
  • Are there specific attributes, structures, or metadata in the Excel file that OutSystems may be sensitive to, which could cause it to fail this conversion?
  • Any suggestions for any scripts, additional tools, checks, or troubleshooting methods to diagnose what might be wrong with the Excel file?

I’m open to any advice or insights you may have. I’ve attached the problematic Excel file for reference.

Unfortunately I didn't make it after the whole week and the hard deadline before the deployment is the 6th September. 

I sincerely appreciate for your help!

Best regards,

  • Yini
response.xlsx
2019-01-07 16-04-16
Siya
 
MVP
Solution

Have followed another approach - copy all the sheets to new excel using the Nuget package "NPOI" .  Not  very efficient approach. Attached the code for your reference. Its works for me. 

code.txt
2019-01-07 16-04-16
Siya
 
MVP

Could you please revise your Azure function code? It seems the issue is in the Python code used to generate the Excel file. Could you try using different libraries for file generation?

Also, when comparing the original file to the converted one, I noticed a difference: the original file has a Byte Order Mark (BOM), but the converted one does not.

2023-02-02 14-34-34
Yini Zhang

Thank you for replying, Siya. 

1. you mean like xlsxwriter instead of openpyxl? The python code was delivered by another team. It's hard for me to try with different approaches and debug. : /

2. So I could also try the BOM removal on the entire Excel file or each XML file by another extension?

2019-01-07 16-04-16
Siya
 
MVP

Please see if you can remove the BOM on the entire excel.

2023-02-02 14-34-34
Yini Zhang

I tried with the following script in different versions, somehow the BOM elements (UTF-16 LE BOM (FF FE) at position 26478 within sheet.xml and UTF-16 BE BOM (FE FF) at position 35684 within the same sheet.xml file. Both are embedded in the XML content, not in Binary Data) could not be removed. Do you have a clue?

public void MssremoveBomFromExcel(byte[] ssinputExcel, out byte[] ssoutputExcel)

    {

        try

        {

            using (MemoryStream outputMemoryStream = new MemoryStream())

            {

                using (MemoryStream inputMemoryStream = new MemoryStream(ssinputExcel))

                {

                    using (ZipArchive inputArchive = new ZipArchive(inputMemoryStream, ZipArchiveMode.Read, true))

                    {

                        using (ZipArchive outputArchive = new ZipArchive(outputMemoryStream, ZipArchiveMode.Create, true))

                        {

                            foreach (var entry in inputArchive.Entries)

                            {

                                var outputEntry = outputArchive.CreateEntry(entry.FullName);


                                using (var inputEntryStream = entry.Open())

                                using (var outputEntryStream = outputEntry.Open())

                                {

                                    byte[] entryBytes = ReadAllBytes(inputEntryStream);

                                    byte[] cleanedBytes;


                                    if (IsTextFile(entry.FullName))

                                    {

                                        string encoding;

                                        cleanedBytes = RemoveBomAndReencode(entryBytes, out encoding);

                                        Console.WriteLine($"Processed {entry.FullName} with encoding {encoding}");

                                    }

                                    else

                                    {

                                        cleanedBytes = entryBytes;

                                    }


                                    outputEntryStream.Write(cleanedBytes, 0, cleanedBytes.Length);

                                }

                            }

                        }

                    }

                }


                ssoutputExcel = outputMemoryStream.ToArray();

            }

        }

        catch (Exception ex)

        {

            throw new Exception("Failed to remove BOM from Excel content.", ex);

        }

    }


    // Method to remove BOM from a byte array and re-encode if necessary

    private static byte[] RemoveBomAndReencode(byte[] bytes, out string encoding)

    {

        encoding = "UTF-8"; // Default encoding


        // Detect and handle UTF-16 LE BOM

        if (bytes.Length >= 2 && bytes[0] == 0xFF && bytes[1] == 0xFE)

        {

            encoding = "UTF-16LE";

            string content = Encoding.Unicode.GetString(bytes.Skip(2).ToArray());

            return Encoding.UTF8.GetBytes(content);

        }

        // Detect and handle UTF-16 BE BOM

        else if (bytes.Length >= 2 && bytes[0] == 0xFE && bytes[1] == 0xFF)

        {

            encoding = "UTF-16BE";

            string content = Encoding.BigEndianUnicode.GetString(bytes.Skip(2).ToArray());

            return Encoding.UTF8.GetBytes(content);

        }

        // Detect and handle UTF-8 BOM

        else if (bytes.Length >= 3 && bytes[0] == 0xEF && bytes[1] == 0xBB && bytes[2] == 0xBF)

        {

            encoding = "UTF-8";

            return bytes.Skip(3).ToArray(); // Already in UTF-8

        }

        // If no BOM detected, return the original byte array

        return bytes;

    }


    // Method to determine if a file is text-based

    private static bool IsTextFile(string entryName)

    {

        return entryName.EndsWith(".xml", StringComparison.OrdinalIgnoreCase) ||

               entryName.EndsWith(".rels", StringComparison.OrdinalIgnoreCase) ||

               entryName.EndsWith(".txt", StringComparison.OrdinalIgnoreCase);

    }


    // Method to read all bytes from a stream

    private static byte[] ReadAllBytes(Stream stream)

    {

        using (MemoryStream ms = new MemoryStream())

        {

            stream.CopyTo(ms);

            return ms.ToArray();

        }

    }


    // MssremoveBomFromExcel

2019-01-07 16-04-16
Siya
 
MVP
Solution

Have followed another approach - copy all the sheets to new excel using the Nuget package "NPOI" .  Not  very efficient approach. Attached the code for your reference. Its works for me. 

code.txt
2023-02-02 14-34-34
Yini Zhang

Indeed! It also worked by my side, the excel file is no longer opened with Protected View and can be directly bootstraped into OutSystems. Many thanks, Siya!

Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.