{"id":241,"date":"2008-03-19T15:37:42","date_gmt":"2008-03-19T23:37:42","guid":{"rendered":"http:\/\/www.scamdex.com\/ScamBlog\/?p=241"},"modified":"2008-04-02T09:06:36","modified_gmt":"2008-04-02T17:06:36","slug":"new-scam-email-indexing-method-again","status":"publish","type":"post","link":"https:\/\/scamdex.com\/ScamBlog\/2008\/03\/new-scam-email-indexing-method-again\/","title":{"rendered":"New Scam Email Indexing Method (again!)"},"content":{"rendered":"

It’s my third iteration on the same basic principle: take a carefully filtered and enhanced archive of 150,000 email messages and then sort, categorize and analyze them, then put them in a defanged, indexable\/searchable list format so that people can browse them.<\/strong><\/p>\n

The first was a program I wrote in perl back in 2004, it was a POP sucker that connected to the mailbox, attempted to extract message parts and rewrite them as a html page. While successful, I was never happy at my efforts to disentangle nested messages and alternate body parts – this meant that a lot of emails showed up with lots of Base64 and other garbage. (eg. ScamDB_S_74.php<\/a>)<\/p>\n

The next try I had was to use a mail archive indexer program called ‘Hypermail<\/a>‘. This was mostly successful at splitting messages into component parts but was still not quite flexible enough for my needs and the indexes were way too long. (eg. HYPMAIL\/date.php<\/a>)<\/p>\n

So this spring, I am trying a whole new system that I rewrote in PHP, my code of choice for the decade. I am still mailbox based, mainly so that I can prune spam that has sneaked through my filters, but that may change soon.<\/p>\n

This is how the Scamdex Engine works:<\/em><\/strong><\/p>\n

    \n
  1. Scam Emails arrive in the honeypot mailbox.<\/li>\n
  2. Using Thunderbird with various Add-ons, I partially manually sort the scam emails into a holding mailstore and throw away the junk.<\/li>\n
  3. A program runs nightly which:\n
      \n
    1. Analyses emails in the holding mailstore into one of 5 categories (419\/AFF, Auctions, Jobs, Phishing, Lottery).<\/li>\n
    2. Adds some extra Headers to the email.<\/li>\n
    3. Moves it to the correct mailbox archive location.<\/li>\n
    4. Runs MHONARC to create the indexed archive and html-ized emails.<\/li>\n
    5. post-processes the MHonarc-ized pages to add a php index include file, update the (MySQL) database and\u00c2\u00a0 distribute the keywords\u00c2\u00a0 and scoring to\u00c2\u00a0 META and the nice little\u00c2\u00a0 graph widget.<\/li>\n
    6. \"Our<\/a><\/li>\n
    7. err… that’s it! <\/a>
      \n<\/a><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

      It’s not pretty or fast but it works, and I can understand it. It’s easy to fix and add to. It’s annoying having to run the process every night from scratch but until I work out how to use the MHONARC system to add\/delete emails from the archive, it’s all I can do. Any suggestions about how I can do this better, let me hear them! <\/em><\/p>\n

      (send to scamblog(a)o7e.net)
      \n<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

      It’s my third iteration on the same basic principle: take a carefully filtered and enhanced archive of 150,000 email messages and then sort, categorize and analyze them, then put them in a defanged, indexable\/searchable list format so that people can browse them. The first was a program I wrote in perl back in 2004, it […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[10,17,12],"tags":[],"_links":{"self":[{"href":"https:\/\/scamdex.com\/ScamBlog\/wp-json\/wp\/v2\/posts\/241"}],"collection":[{"href":"https:\/\/scamdex.com\/ScamBlog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scamdex.com\/ScamBlog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scamdex.com\/ScamBlog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scamdex.com\/ScamBlog\/wp-json\/wp\/v2\/comments?post=241"}],"version-history":[{"count":0,"href":"https:\/\/scamdex.com\/ScamBlog\/wp-json\/wp\/v2\/posts\/241\/revisions"}],"wp:attachment":[{"href":"https:\/\/scamdex.com\/ScamBlog\/wp-json\/wp\/v2\/media?parent=241"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scamdex.com\/ScamBlog\/wp-json\/wp\/v2\/categories?post=241"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scamdex.com\/ScamBlog\/wp-json\/wp\/v2\/tags?post=241"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}