I’ve been working with bugzilla for years.. at Mandriva I was the primary bugzilla care-taker for a few years, and now with Red Hat I do a lot of work on some internal tools that interface with bugzilla that enhances and directs the workflow of day-to-day work (being that we work with bugzilla all the time). I also run my own bugzilla instance a) so I can keep up to date with the goings-on of bugzilla and b) so I can track various things in some of the scripts I write and other stuff. One thing that I do is also use it as a way of logging issues that may come up with the various bits of web hosting that I do.
So what I needed was a way to take incoming emails generated by lfd (my primary concern right now is high load average warnings; I wanted to log them to bugzilla so that if I’m unavailable my dad could also see them (he helps maintain the server and hosts a bunch of his stuff there as well) and perhaps deal with them or at the very least if he wanted/needed to comment on them he could do so via bugzilla, and I can also make note of resolutions or causes, etc. Yes, I’m turning bugzilla into a poor man’s “RT” ticketing system (I have no interest in setting something like that up, I’m already using bugzilla, and this is the best place for me to stuff these sorts of notes). I’ve tried the email_in.pl method and while it works, it only works if you have a specific format so you can assign it to the right component and product — not something that will work with these lfd-generated emails.
Being that a lot of the work that I’ve been doing has to do with using python and xmlrpc to manipulate bugs in the Red Hat bugzilla, it seemed like a reasonable approach to take to deal with my own bug mangling. The problem is that these emails were being sent to root, which in turn forwards directly to me. I also wanted to keep a copy of those mails in my own mailbox in case bugzilla, or anything in between, did something funky, so I opted to do a few things:
- use a gmail filter to forward those emails to another email account specifically for bugzilla mails
- setup fetchmail to pull (via pop3) those emails
- setup procmail to filter those emails and send them to a helper script
- write a helper script that will then call the python-bugzilla tool to file the bug
The first step was easy. Fetchmail was pretty easy too (although it’s been a few years). Procmail was easy, particularly now since I’m only concerned with one particular type of email and my gmail filter is quite specific. The helper script was initially a shell script that was going to call the bugzilla script but I quickly found limitations to that, particular since lfd’s email also has a few attachments and I was having issues with getting it to file the bugs properly. So instead of using uudeview and a shell script, I opted to write something in python.
Instead of procmail feeding uudeview and then feeding my script, I made use of some of the features of python that allow for manipulating email messages (something I’ve never done before). I also found that passing stdin to the python script was somehow also passing stdin to python-bugzilla when I was calling it, which was causing all kinds of grief.
So with this script I learned all kinds of new things: how to manipulate an email message in python and reduce an email with attachments to a message body with individual attachments as objects and how to use subprocess (yes, I’m still using the commands stuff by and large, but that was really problematic with stdin being persistent).
All in all it works quite well. However, I do still have one problem that I’ve not yet liked, and that is with binary files. I’m not sure where the issue is coming from, but for some reason when I call python-bugzilla myself, in a shell, and feed it the file to attach to the bug it works fine — however, when I call it from my script (so no shell), it uses the —file argument as the name of the file and wants stdin as the contents of the file. This is all fine and dandy, but somewhere along the line something is rendering that binary file (was testing with a jpeg) into text and when it’s attached to the bug it has the right mime type, but the contents are wrong and no image is displayed. So, dear lazy web, if there any python folks out there who want to look at my script and tell me what I’m doing wrong, I’d be much obliged…. =)
Anyways, since no post like this is really any good without the files involved, what follows is the script (process-mail) and the .procmailrc file (which is pretty bare bones and doesn’t filter much of anything):
# .procmailrc
#
HOME="/home/mailer"
SHELL=/bin/sh
VERBOSE=off
LOGFILE=$HOME/.procmail/procmail.log
# inserts a blank line between log entries
LOG="
"
:0
*^content-Type:
{
:0fw
| /usr/bin/python $HOME/bin/process-mail
}
All this does is call the process-mail script. It will/may eventually filter on subject and sender if I find that unwanted emails are triggering new bugs. For the moment I don’t particularly care.
And the process-mail python script:
#!/usr/bin/env python
import commands
import email
import os
import subprocess
import sys
import tempfile
# email comes from stdin due to procmail
raw_msg = sys.stdin.readlines()
log = open(os.environ['HOME'] + '/tmp/bugzilla-email.log', 'a')
bz_prog = '/usr/bin/bugzilla'
bz_dest = '--bugzilla=https://bugzilla.annvix.com/xmlrpc.cgi'
directory = tempfile.mkdtemp()
f = tempfile.NamedTemporaryFile(delete=False)
f.write(''.join(raw_msg))
f.close()
msg = email.message_from_file(open(f.name))
attachments = {}
for part in msg.walk():
a_payload = part.get_payload()
a_name = part.get_filename()
a_type = part.get_content_type()
if a_name is None and a_type == 'text/plain':
email_body = part.get_payload()
elif a_name is not None:
tf_name = '%s/%s' % (directory, a_name)
tf_file = open(tf_name, 'wb')
tf_file.write(a_payload)
tf_file.close()
attachments[a_name] = a_type
os.unlink(f.name)
email_to = msg['to']
email_from = msg['from']
email_sub = msg['subject']
log.write('Email received from %s to %s with subject "%s"\n' % (email_from, email_to, email_sub))
cmd = [bz_prog, bz_dest, 'new', '-i', '-p', 'Web Hosting', '-v', 'none', '-c', 'Availability', '-s', email_sub, '-l', email_body]
bug = subprocess.Popen(cmd, stdout=subprocess.PIPE).communicate()[0]
if bug != '':
log.write('Filed bug %s\n' % bug)
if len(attachments) > 0:
for x in attachments:
attachment = os.path.join(directory, x)
log.write('Found attachment: %s\n' % attachment)
cmd = [bz_prog, bz_dest, 'attach', '--file=%s' % x, '--type=%s' % attachments[x], '--desc=mail attachment: %s' % x, bug.strip('\n')]
if 'text' in attachments[x]:
a_file = open(attachment, 'r')
else:
# this does not work.. attaching a jpg results in mangled text and I'm not sure why...
a_file = open(attachment, 'rb')
foo = subprocess.Popen(cmd, stdin=a_file, stdout=subprocess.PIPE).communicate()[0]
a_file.close()
if foo != '':
log.write(foo + '\n')
else:
log.write('Failed to attach bug to bugzilla!\n')
log.close()
sys.exit(0)
I’m not sure why this isn’t working for binary attachments though… it’s probably something simple, but I’ve not had a chance to figure out what the problem is. Dear lazy web…. any advice? =)