When doing automated tasks, recording what is going on and executed is a necessity, so we know if there are any errors and where exactly they occur. For this HackerNews project, we will write a log in txt using a function that we will call from the main file. So, let’s create a function to write log in the customFunctionsGeneral file.
import sharedvars as sv
from dotenv import load_dotenv
import os as os
from sqlalchemy import create_engine
import pandas as pd
import sys
def write_log(message=''):
""" Retrieves message given as parameter and logs execution details and
timestamp in file specified in sharedVars.py. If file hasn't been created,
then create a new file designated to the current time of execution.
"""
# Initiate a file name to write
if sv.file_name_for_log is None:
sv.file_name_for_log = dt.datetime.now().strftime('%Y_%m_%d_%H_%M_%S')
# Check if a text is provided
if message == '':
print('Hey you need to provide a message to write in a log file')
# If provided, we will write text message to the file
else:
with open(f'{sv.file_name_for_log}.txt', 'a') as f:
f.write('On ' + dt.datetime.now().strftime('%b %d, %Y') + ' at ' +
dt.datetime.now().strftime('%H:%M:%S') + ', '
+ message + '\n')
Whenever the write_log function is called by running the main.py file, with specific parameters (message) in my main file, the function will create a new log file and record whatever is put into write_log. Therefore, when the main.py file is run, there should be one write_log file containing all the time and executions of the ETL process.
