Amarisoft

Remote API - Python

 

The purpose of this tutorial is to show to provide a simple python code example by which you can use Amarisoft Remote API.  Motivations for this tutorial are as follows :

NOTE : If you just want to use Remote API without any extention or customization, I would suggest you to use ws.js script as it is since it will be more widely tested and proved working stably for many years. But if you want to revise functionality of ws.js and you are not familiar with Nodejs based Javascript, this tutorial can be a good starting point.

NOTE :  The purpose of this tutorial is to provide proof of concept examples for Python Script to send/recieve Remote API and post processing for the received data. We do not provide any technical support for Python script or troubleshoot Python installation.

 

 

Table of Contents

 

 

Test Setup

 

You can use any kind of test setup that allow you to get UE attached to callbox.

 

 

 

Python and Shell Information

 

Following is the version of the python and package that are used for this tutorial. (I tested this on Windows 11 Home Edition). I don't think there is any specific dependencies for Windows Powershell version and Windows Commad Line Windows version. For python, first try with whatever version you are using as long as it is ver 3.x and upgrade it to latest version if it does not work.

 

 

 

Python Script - Command Line

 

Here I share a super simple python code that can do the function of Remote API. The purpose is to provide you with the most fundamental skeletone of the code in the simplest form and I think this simple code would be more helpful for you to get the big picture of the implementation. Intentionally I didn't put any error trapping or other auxiliary routines for the simplicity. You may extend this code depending on your own necessity.

 

 

Script - Barebone

 

Following is the source script without any comments. I didn't put any comments on purpose for simplicity.

import sys

import json

import argparse

from websocket import create_connection

 

def main():

    parser = argparse.ArgumentParser(description="WebSocket remote API tool")

    parser.add_argument("server", help="Server address")

    parser.add_argument("message", help="JSON message")

    args = parser.parse_args()

 

    ws = create_connection("ws://" + args.server)

    ws.send(args.message)

 

    while True:

        result = ws.recv()

        if result:

            message = json.loads(result)

            print(json.dumps(message, indent=2, ensure_ascii=False))

            if message.get("message") != "ready":

                break

 

    ws.close()

 

if __name__ == "__main__":

    main()

 

 

Script - Comments

 

For the readers who is not familiar with python script and packages that are used in the code. I put the comments for each lines.

# Import the sys module to interact with the Python runtime environment, such as system-specific parameters and functions

import sys

 

# Import the json module to handle JSON data, which allows encoding and decoding JSON data structures

import json

 

# Import the argparse module to handle command-line arguments, which makes it easy to write user-friendly command-line interfaces

import argparse

 

# Import the create_connection function from the websocket module, which is used to establish WebSocket connections

from websocket import create_connection

 

# Define the main function

def main():

 

    # Create an ArgumentParser object with a description, which will hold all the information necessary to parse the command-line arguments

    parser = argparse.ArgumentParser(description="WebSocket remote API tool")

 

    # Add the server argument to the parser, which specifies the WebSocket server address

    parser.add_argument("server", help="Server address")

 

    # Add the message argument to the parser, which specifies the JSON message to be sent to the WebSocket server

    parser.add_argument("message", help="JSON message")

 

    # Parse the command-line arguments and store the parsed arguments as attributes of an object in the args variable

    args = parser.parse_args()

 

    # Create a WebSocket connection to the server using the create_connection function, which takes a WebSocket URL as an argument

    ws = create_connection("ws://" + args.server)

 

    # Send the JSON message to the WebSocket server using the send method of the WebSocket connection object

    ws.send(args.message)

 

    # Continuously receive messages from the WebSocket server

    while True:

 

       # Receive a message from the WebSocket server using the recv method of the WebSocket connection object, which blocks until a message is received

       # The ws.recv() method in the websocket library is a blocking call. This means that it will block the execution of your program until it receives a message from the WebSocket server.

       # When you call ws.recv(), the method waits for a message to arrive from the server. Once a message is received, the method returns the message as a string,

       # and the program execution continues. If no message is received, the method will block indefinitely, and your program will appear to "hang" until a message arrives

       # or the WebSocket connection is closed.

        result = ws.recv()

 

        # If there is a message

        if result:

            # Load the JSON message using the json.loads function, which deserializes a JSON string into a Python object

            # this is used to parse the result string, which is a JSON-formatted message received from the WebSocket server.

            # When json.loads(result) is called, it parses the input JSON string result and returns a Python object (usually a dictionary or a list, depending on the structure of the JSON data).

            # This Python object is then stored in the message variable and can be easily accessed and manipulated using standard Python techniques.

            message = json.loads(result)

 

           # Print the JSON message using the json.dumps function, which serializes a Python object into a JSON formatted string with a pretty format

           # this is used to convert the message object into a JSON-formatted string with specific formatting options.

           #Here's a breakdown of the parameters used in this function call:

           #    message: This is the Python object that you want to convert to a JSON-formatted string. In the provided code, message is the JSON object received from the WebSocket server.

           #    indent=2: This optional parameter is used to define the number of spaces to use as an indentation for the output JSON-formatted string.

           #                   In this case, indent=2 means that the output JSON string will be formatted with a two-space indentation for better readability.

           #                   If the indent parameter is not provided or set to None, the output JSON string will be compact and without any indentation.

           #    ensure_ascii=False: This optional parameter is used to control the encoding of non-ASCII characters in the output JSON-formatted string.

           #                  By default, ensure_ascii is set to True, which means all non-ASCII characters will be escaped using the \uXXXX notation in the output JSON string.

           #                  By setting ensure_ascii=False, non-ASCII characters will be output as-is, without being escaped.

           #                  This can make the output more readable if you expect non-ASCII characters in your JSON data.

            print(json.dumps(message, indent=2, ensure_ascii=False))

 

            # If the message's "message" attribute does not have a value of "ready", break the loop

            if message.get("message") != "ready":

                break

 

    # Close the WebSocket connection using the close method of the WebSocket connection object, which releases the resources associated with the connection

    ws.close()

 

# If the script is run directly (not imported as a module), call the main function

if __name__ == "__main__":

    main()

 

 

 

Script - Test

 

This is how I tested the script. I tested the scrip on a windows PC using Powershell. As you would notice, the string format is a little bit different from the native string format shown in the tutorial Remote API because of the way Windows Powersheel process the string. (NOTE : If you are testing the code in a Linux, you can use the string format as shown in Remote API )

RemoteAPI Python CLI Test 01

NOTE :  For Windows users, you may get errors if you just copy and paste the command as below. Check out JSON String issue on Windows to workaround the problem.

 

Following is the result for the command :

RemoteAPI Python CLI Test 02

 

 

 

Post Processingy - Command Line

 

In this section, I will provide some examples of utility script to post process the log_get output. Before you use this script, you need to do following :

Once you get the output file, you can post process the output file using this script as follows. It is assumed that the python script filename is process_log.py and the file containing the log is log.txt

       $> python process_log.py  log.txt.  

 

Overall Procedure (or Program Structure) is as follows :

RemoteAPI Python ProcessLog 01

 

RemoteAPI Python ProcessLog 02

RemoteAPI Python ProcessLog 03

 

 

Script - Barebone

This script reads log files containing JSON objects and converts them into a Pandas DataFrame. The  script is designed to process log files that have JSON objects embedded within them. The script reads the log file and identifies JSON objects by counting the opening and closing braces,  and then extracts the JSON objects from the text. These JSON objects are then parsed using the Python json library to create Python dictionaries. The dictionaries are further processed and their relevant information is extracted and organized into rows with specific columns that correspond to the  log file attributes. These rows are then used to create a Pandas DataFrame, which is a two-dimensional, size-mutable, and potentially heterogeneous tabular data structure with labeled axes (rows and columns). The DataFrame allows for easy manipulation, analysis, and export of the log data in various formats, such as HTML tables in this case.

 

import json

import pandas as pd

import sys

import re

import ijson

import chardet

import html

import datetime

 

 

def find_json_objects(text):

    json_objects = []

    open_braces = 0

    json_start = -1

 

    for idx, char in enumerate(text):

        if char == '{':

            if open_braces == 0:

                json_start = idx

            open_braces += 1

        elif char == '}':

            open_braces -= 1

            if open_braces == 0:

                json_objects.append(text[json_start:idx+1])

 

    return json_objects

 

 

def log2panda(log_filename: str):

    column_names = ['src', 'idx', 'level', 'timestamp', 'layer', 'dir', 'ue_id', 'cell', 'rnti', 'frame', 'slot', 'channel', 'data']

    data = []

 

    # Detect the file encoding

    with open(log_filename, 'rb') as log_file:

        encoding_result = chardet.detect(log_file.read())

    file_encoding = encoding_result['encoding']

 

    # Open the file with the detected encoding

    with open(log_filename, 'r', encoding=file_encoding) as log_file:

        content = log_file.read()

        json_objects = find_json_objects(content)

 

        for json_object in json_objects:

            try:

                log_json = json.loads(json_object)

 

                if "logs" in log_json:

                    logs = log_json["logs"]

                else:

                    logs = [log_json]

 

                for log in logs:

                    if log.get("message") != "ready":

                        row = []

                        for column in column_names:

                            cell_value = log.get(column)

                            if column == 'data' and isinstance(cell_value, list):

                                cell_value = '\n'.join(cell_value)

                            if cell_value is not None:

                                if column == 'data':

                                    cell_value = cell_value.replace('\\r\\n', '<br>&nbsp;;')

                                    cell_value = cell_value.replace('\\t', '&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;')

                                cell_value = str(cell_value).replace('"', '')

                            else:

                                cell_value = ' '

                            row.append(cell_value)

                        data.append(row)

            except json.JSONDecodeError:

                continue

 

    df = pd.DataFrame(data, columns=column_names)

    return df

 

 

def panda2html(df, html_filename):

    with open(html_filename, 'w', encoding='utf-8') as f:

        f.write('<!DOCTYPE html>\n')

        f.write('<html>\n')

        f.write('<head>\n')

        f.write('<style>\n')

        f.write('table { font-family: Arial, sans-serif; font-size: 10px; }\n')

        f.write('</style>\n')

        f.write('</head>\n')

        f.write('<body>\n')

 

        f.write('<table border="1" cellspacing="0" cellpadding="5">\n')

        

        # Write the header row

        f.write('<thead>\n')

        f.write('<tr>')

        f.write('<th style="vertical-align: top;">Line #</th>')  # Add 'Line #' column header

        for column in df.columns:

            if column == 'data':

                f.write(f'<th style="width:50%; vertical-align: top;">{column}</th>')

            else:

                f.write(f'<th style="vertical-align: top;">{column}</th>')

        f.write('</tr>\n')

        f.write('</thead>\n')

 

        # Write the data rows

        f.write('<tbody>\n')

        for row_num, row in enumerate(df.iterrows(), start=1):

            f.write('<tr>')

            f.write(f'<td style="vertical-align: top;">{row_num}</td>')  # Add row number

            for index, cell in enumerate(row[1]):

                if index == df.columns.get_loc('timestamp'):

                    timestamp = datetime.datetime.fromtimestamp(int(cell) / 1000)

                    formatted_timestamp = timestamp.strftime("%H:%M:%S.%f")[:-3]

                    f.write(f'<td style="vertical-align: top;">{formatted_timestamp}</td>')

                elif index == df.columns.get_loc('data'):

                    cell = cell.replace('\\r\\n', '\n')

                    cell = cell.replace('\\t', '    ')

                    f.write(f'<td style="vertical-align: top;"><pre>{html.escape(cell)}</pre></td>')

                else:

                    f.write(f'<td style="vertical-align: top;">{html.escape(cell)}</td>')

            f.write('</tr>\n')

        f.write('</tbody>\n')

 

        f.write('</table>')

 

        f.write('</body>\n')

        f.write('</html>\n')

 

 

def panda2html_sheet(df, html_filename):

    with open(html_filename, 'w', encoding='utf-8') as f:

        f.write('<!DOCTYPE html>\n')

        f.write('<html>\n')

        f.write('<head>\n')

        #f.write('<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.11.5/css/jquery.dataTables.min.css">\n')

        #f.write('<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/searchbuilder/1.3.1/css/searchBuilder.dataTables.min.css">\n')

        #f.write('<script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>\n')

        #f.write('<script type="text/javascript" src="https://cdn.datatables.net/1.11.5/js/jquery.dataTables.min.js"></script>\n')

        #f.write('<script type="text/javascript" src="https://cdn.datatables.net/searchbuilder/1.3.1/js/dataTables.searchBuilder.min.js"></script>\n')

        f.write('<link rel="stylesheet" type="text/css" href="css/jquery.dataTables.min.css">\n')

        f.write('<link rel="stylesheet" type="text/css" href="css/searchBuilder.dataTables.min.css">\n')

        f.write('<script src="js/jquery-3.6.0.min.js"></script>\n')

        f.write('<script type="text/javascript" src="js/jquery.dataTables.min.js"></script>\n')

        f.write('<script type="text/javascript" src="js/dataTables.searchBuilder.min.js"></script>\n')

        f.write('<style>\n')

        f.write('table { font-family: Arial, sans-serif; font-size: 10px; }\n')

        f.write('</style>\n')

        f.write('</head>\n')

        f.write('<body>\n')

 

        f.write('<table id="data-table" border="1" cellspacing="0" cellpadding="5">\n')

 

        # Write the header row

        f.write('<thead>\n')

        f.write('<tr>')

        f.write('<th style="vertical-align: top;">Line #</th>')  # Add 'Line #' column header

        for column in df.columns:

            if column == 'data':

                f.write(f'<th style="width:50%; vertical-align: top;">{column}</th>')

            else:

                f.write(f'<th style="vertical-align: top;">{column}</th>')

        f.write('</tr>\n')

        f.write('</thead>\n')

 

        # Write the data rows

        f.write('<tbody>\n')

        for row_num, row in enumerate(df.iterrows(), start=1):

            f.write('<tr>')

            f.write(f'<td style="vertical-align: top;">{row_num}</td>')  # Add row number

            for index, cell in enumerate(row[1]):

                if index == df.columns.get_loc('timestamp'):

                    timestamp = datetime.datetime.fromtimestamp(int(cell) / 1000)

                    formatted_timestamp = timestamp.strftime("%H:%M:%S.%f")[:-3]

                    f.write(f'<td style="vertical-align: top;">{formatted_timestamp}</td>')

                elif index == df.columns.get_loc('data'):

                    cell = cell.replace('\\r\\n', '\n')

                    cell = cell.replace('\\t', '    ')

                    f.write(f'<td style="vertical-align: top;"><pre>{html.escape(cell)}</pre></td>')

                else:

                    f.write(f'<td style="vertical-align: top;">{html.escape(cell)}</td>')

            f.write('</tr>\n')

        f.write('</tbody>\n')

        f.write('</table>\n')

 

        f.write('<script>\n')

        f.write('$(document).ready(function() {\n')

        f.write('  var table = $("#data-table").DataTable({\n')

        f.write('    searchBuilder: {\n')

        f.write('      columns: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] // Add more column indexes if you have more columns\n')

        f.write('    },\n')

        f.write('    pageLength: -1, // Set the default number of rows to display to "All"\n')

        f.write('    lengthMenu: [[10, 25, 50, -1], [10, 25, 50, "All"]], // Define the options for the "Show entries" dropdown\n')

        f.write('    dom: "Qlfrtip" // Add "Q" to the "dom" parameter to enable the searchBuilder extension\n')

        f.write('  });\n')

        f.write('});\n')

        f.write('</script>\n')

 

        f.write('</body>\n')

        f.write('</html>\n')

 

 

 

 

def main(log_filename: str):

    df = log2panda(log_filename)

    #print(df.head(20))

    #html_filename = 'output.html'

    #panda2html(df, html_filename)

    html_filename = 'output_sheet.html'

    panda2html_sheet(df, html_filename)

 

if __name__ == "__main__":

    log_filename = sys.argv[1]

    main(log_filename)

 

 

 

Script - Comment

 

#    You need to install following packages to run this code      

#           pip install pandas

#           pip install ijson

#           pip install chardet

#           NOTE : json, sys, re, html, datetime is included in default package

#

 

import json

import pandas as pd

import sys

import re

import ijson

import chardet

import html

import datetime

 

# The find_json_objects(text) function is a utility function that takes a string input, text, which represents the content of a log file. The purpose of this  function is to identify and extract all JSON

# objects present within the text.

# To achieve this, the function initializes two variables: json_objects, an empty list that will store the extracted JSON objects, and open_braces, a counter  that will be used to track the opening and

#  closing braces of the JSON objects.

# The function iterates through the input text character by character, using a for loop. For each character, it checks if the character is an opening brace ({)  or a closing brace (}). If it encounters an

# opening brace, it increments the open_braces counter and, if it's the first opening brace, it stores the current index as the starting position of the JSON object. If it encounters a closing brace, it

# decrements the open_braces counter. When the counter reaches zero,  it means that a complete JSON object has been identified, so it extracts the substring from the starting position to the current

#  index and appends it to the json_objects list.

# Once the loop is finished, the function returns the json_objects list, which contains all the extracted JSON objects from the input text as strings.

 

def find_json_objects(text):

    json_objects = []

    open_braces = 0

    json_start = -1

 

    for idx, char in enumerate(text):

        if char == '{':

            if open_braces == 0:

                json_start = idx

            open_braces += 1

        elif char == '}':

            open_braces -= 1

            if open_braces == 0:

                json_objects.append(text[json_start:idx+1])

 

    return json_objects

 

 

# The log2panda(log_filename: str) function is responsible for processing a log file containing JSON objects and converting the contents into a Pandas DataFrame.

# The function takes a single argument, log_filename, which is a string representing the name of the log file to be processed.

# Here's a step-by-step description of what the function does:

# 1. It initializes the column_names list, which defines the columns for the DataFrame, and an empty data list to store the rows of the DataFrame.

# 2. It opens the log file with the 'rb' (read binary) mode and uses the chardet library to detect the file's encoding. The detected encoding is stored in the

#    file_encoding variable.

# 3. It opens the log file again, this time using the detected encoding, and reads the entire content into the content variable.

# 4. It calls the find_json_objects(content) function to extract all JSON objects from the file content.

# 5. It iterates through the extracted JSON objects, parsing each JSON object into a Python dictionary using the json.loads() function. It checks if the dictionary

#    contains a "logs" key, which represents multiple logs. If so, it processes each log in the list. Otherwise, it processes the single log represented by the dictionary.

# 6. It skips logs with a "message" value equal to "ready" and iterates through the remaining logs. For each log, it creates a new row, extracting the values for

#    each column defined in column_names and appending them to the row. The rows are then appended to the data list.

# 7. In case of a JSON decoding error, it continues to the next JSON object without processing the current one.

# 8. Once all logs are processed, it creates a Pandas DataFrame using the data list and the column_names list, and returns the DataFrame.

 

def log2panda(log_filename: str):

    column_names = ['src', 'idx', 'level', 'timestamp', 'layer', 'dir', 'ue_id', 'cell', 'rnti', 'frame', 'slot', 'channel', 'data']

    data = []

 

    # Detect the file encoding

    with open(log_filename, 'rb') as log_file:

        encoding_result = chardet.detect(log_file.read())

    file_encoding = encoding_result['encoding']

 

    # Open the file with the detected encoding

    with open(log_filename, 'r', encoding=file_encoding) as log_file:

        content = log_file.read()

        json_objects = find_json_objects(content)

 

        for json_object in json_objects:

            try:

                log_json = json.loads(json_object)

 

                if "logs" in log_json:

                    logs = log_json["logs"]

                else:

                    logs = [log_json]

 

                for log in logs:

                    if log.get("message") != "ready":

                        row = []

                        for column in column_names:

                            cell_value = log.get(column)

                            if column == 'data' and isinstance(cell_value, list):

                                cell_value = '\n'.join(cell_value)

                            if cell_value is not None:

                                if column == 'data':

                                    cell_value = cell_value.replace('\\r\\n', '<br>&nbsp;;')

                                    cell_value = cell_value.replace('\\t', '&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;')

                                cell_value = str(cell_value).replace('"', '')

                            else:

                                cell_value = ' '

                            row.append(cell_value)

                        data.append(row)

            except json.JSONDecodeError:

                continue

 

    df = pd.DataFrame(data, columns=column_names)

    return df

 

 

 

# The panda2html(df, html_filename) function takes a Pandas DataFrame (df) and an output HTML file name (html_filename) as input arguments. It converts the contents  of the DataFrame  

# into a neatly formatted HTML table and writes it to the specified file

# 1. It opens the output HTML file with write ('w') mode and the 'utf-8' encoding.

# 2. It writes the required HTML tags, such as <!DOCTYPE html>, <html>, <head>, and <body> to the output file. It also includes a <style> tag to define some basic

#    CSS styles for the table, such as font family and font size.

# 3. It starts creating the table by writing the opening <table> tag with appropriate attributes like border, cellspacing, and cellpadding.

# 4. It writes the table header by creating a new row (<tr>), iterating through the columns of the DataFrame, and writing a table header cell (<th>) for each column.

#    The header row is wrapped with <thead> tags.

# 5. It writes the table data rows by iterating through the rows of the DataFrame. For each row, it creates a new row (<tr>) in the HTML table and iterates through

#    the cell values of the DataFrame row, writing a table data cell (<td>) for each cell value. The data rows are wrapped with <tbody> tags.

#        * It adds the row number as the first cell in each row.

#        * For the 'timestamp' column, it converts the Unix timestamp to a formatted human-readable timestamp before writing it to the cell.

#        * For the 'data' column, it replaces '\r\n' with '\n' and '\t' with ' ' before writing the cell content using the <pre> tag to preserve the original formatting.

#        * For other columns, it writes the cell values as is, using the html.escape() function to prevent any HTML injection issues.

# 6. It closes the table by writing the closing </table> tag.

# 7. It writes the closing </body> and </html> tags to the output file and closes it.

 

def panda2html(df, html_filename):

    with open(html_filename, 'w', encoding='utf-8') as f:

        f.write('<!DOCTYPE html>\n')

        f.write('<html>\n')

        f.write('<head>\n')

        f.write('<style>\n')

        f.write('table { font-family: Arial, sans-serif; font-size: 10px; }\n')

        f.write('</style>\n')

        f.write('</head>\n')

        f.write('<body>\n')

 

        f.write('<table border="1" cellspacing="0" cellpadding="5">\n')

        

        # Write the header row

        f.write('<thead>\n')

        f.write('<tr>')

        f.write('<th style="vertical-align: top;">Line #</th>')  # Add 'Line #' column header

        for column in df.columns:

            if column == 'data':

                f.write(f'<th style="width:50%; vertical-align: top;">{column}</th>')

            else:

                f.write(f'<th style="vertical-align: top;">{column}</th>')

        f.write('</tr>\n')

        f.write('</thead>\n')

 

        # Write the data rows

        f.write('<tbody>\n')

        for row_num, row in enumerate(df.iterrows(), start=1):

            f.write('<tr>')

            f.write(f'<td style="vertical-align: top;">{row_num}</td>')  # Add row number

            for index, cell in enumerate(row[1]):

                if index == df.columns.get_loc('timestamp'):

                    timestamp = datetime.datetime.fromtimestamp(int(cell) / 1000)

                    formatted_timestamp = timestamp.strftime("%H:%M:%S.%f")[:-3]

                    f.write(f'<td style="vertical-align: top;">{formatted_timestamp}</td>')

                elif index == df.columns.get_loc('data'):

                    cell = cell.replace('\\r\\n', '\n')

                    cell = cell.replace('\\t', '    ')

                    f.write(f'<td style="vertical-align: top;"><pre>{html.escape(cell)}</pre></td>')

                else:

                    f.write(f'<td style="vertical-align: top;">{html.escape(cell)}</td>')

            f.write('</tr>\n')

        f.write('</tbody>\n')

 

        f.write('</table>')

 

        f.write('</body>\n')

        f.write('</html>\n')

 

 

# The panda2html_sheet(df, html_filename) function takes a Pandas DataFrame (df) and an output HTML file name (html_filename) as input arguments. It converts the contents of the DataFrame

# into a searchable and sortable HTML table using the DataTables jQuery plugin and writes it to the specified file.

# 1. It opens the output HTML file with write ('w') mode and the 'utf-8' encoding.

# 2. It writes the required HTML tags, such as <!DOCTYPE html>, <html>, <head>, and <body> to the output file. It also includes a <style> tag to define some basic

#    CSS styles for the table, such as font family and font size.

# 3. It writes the required DataTables CSS and JavaScript library links and scripts, as well as the custom DataTables configurations inside the <head> tag.

# 4. It starts creating the table by writing the opening <table> tag with an id attribute set to "data-table" and appropriate attributes like border, cellspacing,

#    and cellpadding.

# 5. It writes the table header by creating a new row (<tr>), iterating through the columns of the DataFrame, and writing a table header cell (<th>) for each column.

#    The header row is wrapped with <thead> tags.

# 6. It writes the table data rows by iterating through the rows of the DataFrame. For each row, it creates a new row (<tr>) in the HTML table and iterates through

#    the cell values of the DataFrame row, writing a table data cell (<td>) for each cell value. The data rows are wrapped with <tbody> tags.

#        * It adds the row number as the first cell in each row.

#        * For the 'timestamp' column, it converts the Unix timestamp to a formatted human-readable timestamp before writing it to the cell.

#        * For the 'data' column, it replaces '\r\n' with '\n' and '\t' with ' ' before writing the cell content using the <pre> tag to preserve the original formatting.

#        * For other columns, it writes the cell values as is, using the html.escape() function to prevent any HTML injection issues.

# 7. It closes the table by writing the closing </table> tag.

# 8. It writes a <script> tag to initialize the DataTables plugin on the created table, with custom configurations such as enabling the searchBuilder extension,

#    setting the default number of rows to display to "All", and defining the options for the "Show entries" dropdown.

# 9. It writes the closing </body> and </html> tags to the output file and closes it.

 

def panda2html_sheet(df, html_filename):

    with open(html_filename, 'w', encoding='utf-8') as f:

        f.write('<!DOCTYPE html>\n')

        f.write('<html>\n')

        f.write('<head>\n')

        f.write('<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.11.5/css/jquery.dataTables.min.css">\n')

        f.write('<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/searchbuilder/1.3.1/css/searchBuilder.dataTables.min.css">\n')

        f.write('<script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>\n')

        f.write('<script type="text/javascript" src="https://cdn.datatables.net/1.11.5/js/jquery.dataTables.min.js"></script>\n')

        f.write('<script type="text/javascript" src="https://cdn.datatables.net/searchbuilder/1.3.1/js/dataTables.searchBuilder.min.js"></script>\n')

        # f.write('<link rel="stylesheet" type="text/css" href="css/jquery.dataTables.min.css">\n')

        # f.write('<link rel="stylesheet" type="text/css" href="css/searchBuilder.dataTables.min.css">\n')

        # f.write('<script src="js/jquery-3.6.0.min.js"></script>\n')

        #f.write('<script type="text/javascript" src="js/jquery.dataTables.min.js"></script>\n')

        #f.write('<script type="text/javascript" src="js/dataTables.searchBuilder.min.js"></script>\n')

        f.write('<style>\n')

        f.write('table { font-family: Arial, sans-serif; font-size: 10px; }\n')

        f.write('</style>\n')

        f.write('</head>\n')

        f.write('<body>\n')

 

        f.write('<table id="data-table" border="1" cellspacing="0" cellpadding="5">\n')

 

        # Write the header row

        f.write('<thead>\n')

        f.write('<tr>')

        f.write('<th style="vertical-align: top;">Line #</th>')  # Add 'Line #' column header

        for column in df.columns:

            if column == 'data':

                f.write(f'<th style="width:50%; vertical-align: top;">{column}</th>')

            else:

                f.write(f'<th style="vertical-align: top;">{column}</th>')

        f.write('</tr>\n')

        f.write('</thead>\n')

 

        # Write the data rows

        f.write('<tbody>\n')

        for row_num, row in enumerate(df.iterrows(), start=1):

            f.write('<tr>')

            f.write(f'<td style="vertical-align: top;">{row_num}</td>')  # Add row number

            for index, cell in enumerate(row[1]):

                if index == df.columns.get_loc('timestamp'):

                    timestamp = datetime.datetime.fromtimestamp(int(cell) / 1000)

                    formatted_timestamp = timestamp.strftime("%H:%M:%S.%f")[:-3]

                    f.write(f'<td style="vertical-align: top;">{formatted_timestamp}</td>')

                elif index == df.columns.get_loc('data'):

                    cell = cell.replace('\\r\\n', '\n')

                    cell = cell.replace('\\t', '    ')

                    f.write(f'<td style="vertical-align: top;"><pre>{html.escape(cell)}</pre></td>')

                else:

                    f.write(f'<td style="vertical-align: top;">{html.escape(cell)}</td>')

            f.write('</tr>\n')

        f.write('</tbody>\n')

        f.write('</table>\n')

 

        f.write('<script>\n')

        f.write('$(document).ready(function() {\n')

        f.write('  var table = $("#data-table").DataTable({\n')

        f.write('    searchBuilder: {\n')

        f.write('      columns: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] // Add more column indexes if you have more columns\n')

        f.write('    },\n')

        f.write('    pageLength: -1, // Set the default number of rows to display to "All"\n')

        f.write('    lengthMenu: [[10, 25, 50, -1], [10, 25, 50, "All"]], // Define the options for the "Show entries" dropdown\n')

        f.write('    dom: "Qlfrtip" // Add "Q" to the "dom" parameter to enable the searchBuilder extension\n')

        f.write('  });\n')

        f.write('});\n')

        f.write('</script>\n')

 

        f.write('</body>\n')

        f.write('</html>\n')

 

 

 

def main(log_filename: str):

    df = log2panda(log_filename)

    #print(df.head(20))

    #html_filename = 'output.html'

    #panda2html(df, html_filename)

    html_filename = 'output_sheet.html'

    panda2html_sheet(df, html_filename)

 

if __name__ == "__main__":

    log_filename = sys.argv[1]

    main(log_filename)

 

 

 

 

 

© 2024 All rights reserved | Legal notice |